Grafana OnCall: Connect to Discord, Mattermost, and more with webhooks

One important consideration when adopting a tool is whether it can integrate with your existing workflows and services. Each scenario can be highly specific, which is why it’s important to look for tools that have a public API or customizable webhooks.

Last year, Grafana OnCall expanded its webhook support to allow for more complex setups, offering greater flexibility to interact with other services during alert group events.

How do these webhooks work? To illustrate how you can adapt Grafana OnCall and tie it to your existing environment, let’s show you how to send alert group notifications and updates to other messaging solutions like Mattermost and Discord, which are not yet available out of the box but can still be plugged into your workflow.

The goal here will be to connect Grafana OnCall webhooks to a third-party API, establishing a simple one-way sync (OnCall -> external service) to keep status information current without writing any code. We’ll walk through the process for each popular ChatOps app individually to show that while the connections may be different, the setup is essentially the same within Grafana OnCall, so let’s see how it works!

Note: For a more complete integration or a two-way sync, additional logic and/or use our OnCall API will be necessary.

How to integrate Grafana OnCall with Mattermost

Mattermost is a popular open source and flexible messaging service supporting multiple integrations that you can self-host or use as cloud as a service. For this integration, the plan is to create a bot that will post alert group details to a Mattermost channel and keep those alert groups’ status and information up to date.

First, let’s create the bot account to access the Mattermost API using its access token:

Note: you may need to set the System Admin role to allow the bot to update the messages. See this Mattermost FAQ for more information.

You will get a token upon creation; make sure to keep that value at hand.

Next, add the bot user as a member in the channel you want the notifications posted to, and get the channel ID from that channel’s information view. To get there, click on the channel name dropdown and you will see the “View Info” option. 

At this point you should have a token and the ability to use that token, as well as the channel ID, to hit the Mattermost API to post a message in the indicated Mattermost channel as the bot. Use this curl command to test the flow we want to use in Grafana OnCall:

Copy

$ curl -i -X POST -H 'Content-Type: application/json' -d '{"channel_id":"eorfq3i1zj89jg7pqpzpyp8rxh", "message":"This is a message from a bot", "props":{"attachments": [{"pretext": "Look some text","text": "This is text"}]}}' -H 'Authorization: Bearer qwd4k954qjyx3euumnbnqfa6wo' http://localhost:8065/api/v4/posts8065/api/v4/posts

Which should return an output similar to the following:

Copy

HTTP/1.1 201 Created
Content-Type: application/json
Permissions-Policy:
Referrer-Policy: no-referrer
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Request-Id: mdubxdeqffffpxkcbu65fxaino
X-Version-Id: 7.9.0.7.9.0.7f5ade118b6f3cf753675a3fdfaa6d40.false
Date: Tue, 16 Apr 2024 17:11:46 GMT
Content-Length: 725

{"id":"kxi7owmor3y9proo5rkef64our","create_at":1713287506329,"update_at":1713287506329,"edit_at":0,"delete_at":0,"is_pinned":false,"user_id":"rks6to6tz7n19etm6oboe9s9zh","channel_id":"eorfq3i1zj89jg7pqpzpyp8rxh","root_id":"","original_id":"","message":"This is a message from a bot","type":"","props":{"attachments":[{"id":0,"fallback":"","color":"","pretext":"Look some text","author_name":"","author_link":"","author_icon":"","title":"","title_link":"","text":"This is text","fields":null,"image_url":"","thumb_url":"","footer":"","footer_icon":"","ts":null}],"from_bot":"true"},"hashtags":"","pending_post_id":"","reply_count":0,"last_reply_at":0,"participants":null,"metadata":{"embeds":[{"type":"message_attachment"}]}}

Once you make the request above, you should get a message in your Mattermost channel similar to the one in the screenshot below that confirms credentials and channel ID are OK, and that we can use the API.

Going back to our goal of creating a Grafana OnCall integration plan using webhooks, we are interested in the following Mattermost API endpoints (for reference) for our use case:

• Create a post, to post a new alert group notification
• Update a post, to update a previously sent notification when the alert group is updated

How to connect Grafana OnCall to Mattermost

In Grafana OnCall, we will need to configure two webhooks: one to post the notification when an alert group is created (using the “create a post” endpoint), and another to update that notification when the alert group status changes (using the “update a post” endpoint).

Let’s set up the Alert Group Created webhook first:

If you’re following along on your own machine, here is what you need to enter to achieve what you see in the steps outlined in the GIF above:

• Name: mattermost-new
• Enabled: True
• Trigger type: Alert group created
• HTTP method: POST
• Webhook URL: http://localhost:8065/api/v4/posts
• Authorization header: Bearer [your token]
• Forward all: False
• Data:

Copy

{
    "channel_id": "eorfq3i1zj89jg7pqpzpyp8rxh",
    "message": "{% if alert_group.state == 'acknowledged'%}:large_orange_circle:{% elif alert_group.state == 'resolved'%}:large_green_circle:{% elif alert_group.state == 'silenced'%}:white_circle:{% else %}:red_circle:{% endif %} **{{ alert_group.title }}**\n*{{ alert_group.state }}*\n{{ alert_payload.message }}\n*{{ integration.name }}*\n\n{% if event.type == 'acknowledge' %}**Acknowledged by: {{ user.username }}**{% endif %}{% if event.type == 'resolve' %}**Resolved by: {{ user.username }}**{% endif %}{% if event.type == 'silence' %}**Silenced by: {{ user.username }} (until {{ event.until }})**{% endif %}\n\n[View in Grafana OnCall]({{ alert_group.permalinks.web }})"
}

To update a sent notification, you need to have the original message ID. This ID is included in the response after triggering the webhook we just created. Grafana OnCall webhooks maintain a record of previous responses linked to an alert group, allowing us to reference this information in our update webhook. 

Then, we can use the template expression {{ responses.WHK9SLCGJBPZK5.id }} to reference the id field from the previous response to the webhook with ID WHK9SLCGJBPZK5, corresponding to the same alert group triggering the status change. (This is the mattermost-new webhook. You can get this ID from the webhooks listing UI; see below.)

With that in mind, we will set up the Alert Group Status Change webhook now. Here is the information you’ll need to enter to configure your webhook: 

• Name: mattermost-update
• Enabled: True
• Trigger type: Alert group status change
• HTTP method: PUT
• Webhook URL: (Use your mattermost-new webhook ID instead.) http://localhost:8065/api/v4/posts/{{ responses.WHK9SLCGJBPZK5.id }}
• Authorization header: Bearer [your token]
• Forward all: False
• Data: (Use your mattermost-new webhook ID instead.)

Copy

{
    "id": "{{ responses.WHK9SLCGJBPZK5.id }}",
    "message": "{% if alert_group.state == 'acknowledged'%}:large_orange_circle:{% elif alert_group.state == 'resolved'%}:large_green_circle:{% elif alert_group.state == 'silenced'%}:white_circle:{% else %}:red_circle:{% endif %} **{{ alert_group.title }}**\n*{{ alert_group.state }}*\n{{ alert_payload.message }}\n*{{ integration.name }}*\n\n{% if event.type == 'acknowledge' %}**Acknowledged by: {{ user.username }}**{% endif %}{% if event.type == 'resolve' %}**Resolved by: {{ user.username }}**{% endif %}{% if event.type == 'silence' %}**Silenced by: {{ user.username }} (until {{ event.until }})**{% endif %}\n\n[View in Grafana OnCall]({{ alert_group.permalinks.web }})"
}

And that’s it! If you send a demo alert, you should get a notification similar to this one:

Which, after acknowledging, will be updated to:

Note: Since we didn’t set any integrations to filter by, webhooks will be triggered for every alert group in your stack.

How to integrate Grafana OnCall with Discord

Discord is an instant messaging and VoIP social platform that supports voice calls, video calls, text messaging, and media and files. For this integration, the plan is to create a bot that will post alert group details to a channel and keep those alert groups’ status and information up to date.

In this case we will use Discord webhooks to implement the notifications:

Through a webhook URL we can do multiple operations. For our purposes, we are interested in:

• Creating a message, to post a new alert group notification
• Editing a message, to update a previously sent notification when the alert group is updated

Note: when creating a message we will need to set the ?wait=true query param to get a response with the created message information, which we will need to update that message later.

How to connect Grafana OnCall to Discord

Similar to what we did for Mattermost, we need to create the Grafana OnCall webhooks following the linked API’s specification:

Here is what you need to enter to achieve what you see in the steps outlined in the GIF above:

• Name: discord-new
• Enabled: True
• Trigger type: Alert group created
• HTTP method: POST
• Webhook URL: [your Discord webhook URL]?wait=true
• Forward all: False
• Data:

Copy

{
"content": "{% if alert_group.state == 'acknowledged'%}:orange_circle:{% elif alert_group.state == 'resolved'%}:green_circle:{% elif alert_group.state == 'silenced'%}:white_circle:{% else %}:red_circle:{% endif %} **{{ alert_group.title }}**\n*{{ alert_group.state }}*\n{{ alert_payload.message }}\n*{{ integration.name }}*\n\n{% if event.type == 'acknowledge' %}**Acknowledged by: {{ user.username }}**{% endif %}{% if event.type == 'resolve' %}**Resolved by: {{ user.username }}**{% endif %}{% if event.type == 'silence' %}**Silenced by: {{ user.username }} (until {{ event.until }})**{% endif %}\n\n[View in Grafana OnCall]({{ alert_group.permalinks.web }})"
}

For the update webhook, we will use the message id from the previous discord-new response in the URL path:

• Name: discord-update
• Enabled: True
• Trigger type: Alert group status change
• HTTP method: PATCH
• Webhook URL: (Use your discord-new webhook ID instead.) [your Discord webhook URL]/messages/{{ responses.WH5PXWNJ7MT9Y5.id }}
• Forward all: False
• Data:

Copy

{
"content": "{% if alert_group.state == 'acknowledged'%}:orange_circle:{% elif alert_group.state == 'resolved'%}:green_circle:{% elif alert_group.state == 'silenced'%}:white_circle:{% else %}:red_circle:{% endif %} **{{ alert_group.title }}**\n*{{ alert_group.state }}*\n{{ alert_payload.message }}\n*{{ integration.name }}*\n\n{% if event.type == 'acknowledge' %}**Acknowledged by: {{ user.username }}**{% endif %}{% if event.type == 'resolve' %}**Resolved by: {{ user.username }}**{% endif %}{% if event.type == 'silence' %}**Silenced by: {{ user.username }} (until {{ event.until }})**{% endif %}\n\n[View in Grafana OnCall]({{ alert_group.permalinks.web }})"
}

Let’s trigger a demo alert and see how the notification looks:

Which is then updated on acknowledgement:

Set up webhooks programmatically

As a bonus, I have produced a couple scripts to create/update the OnCall webhooks using the Grafana OnCall API. (Did we mention it is useful to have an API?) These scripts make it simpler to iterate on the notification template until you get the desired output. You should be able to do a similar setup using Terraform.

• Script for Mattermost
• Script for Discord

Going further

It could be feasible to follow a similar approach with other ChatOps services like Microsoft Teams or Google Chat. Let us know if you try!

Another enhancement worth exploring is the ability to notify a user according to OnCall schedules. This may be achievable using an Escalation Step webhook in conjunction with the information from the users_to_notify field.

In addition, a more comprehensive integration for Mattermost and Discord is on our radar, and we would be happy to help anyone willing to contribute moving that forward. Feel free to reach out to us on our Community Slack.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!