How to connect ServiceNow to Grafana Cloud IRM incidents

Companies rely on a variety of services to streamline their workflows, which often requires data synchronization or information sharing across platforms. But are your tools flexible enough to connect with external systems?

ServiceNow is widely recognized for its robust and complex workflow support for enterprises. However, it may not always offer the most intuitive or user-friendly experience when handling incidents. That’s why we’ve updated Grafana Cloud IRM to give you one place to manage your incidents and ensure your ServiceNow instance stays up-to-date.

We recently released a new way to set up webhooks based on IRM incident events so that you can now template URLs, headers, and data, as well as chain different events by combining data from previous responses.

With that in mind, and to show an example of how this works, let’s implement a bidirectional status sync between IRM incidents and ServiceNow incidents.

Webhooks for incidents

IRM outgoing webhooks now support incident events, providing a unified experience for automating alert group and incident workflows. You can trigger requests based on key incident lifecycle events, such as when an incident is declared, updated, or resolved.

What does advanced mean in this context? With this updated feature you can:

• Customize webhook requests with support for any HTTP method
• Use templated URLs, headers, and request bodies
• Dynamically reference incident data and prior webhook responses
• View webhook execution details directly from the incident or alert group timeline
• Manage incident-related webhook configurations using Terraform

From IRM to ServiceNow

As a first step, let’s make sure there is a grafana-irm user configured in ServiceNow, as described in our documentation for the IRM ServiceNow integration for alert groups.

To create a ServiceNow incident when an IRM incident is declared, and to set it to closed when incident is resolved, we are going to define a couple of IRM incident webhooks:

Incident declared

Name: Incident declared

Trigger type: Incident Declared

HTTP method: POST

Webhook URL: https://devXXXXXX.service-now.com/api/now/table/incident

Username: grafana-irm

Password: [grafana-irm-snow-password]

Trigger template:

Copy

{{ incident.createdByUser.name.lower() != "service account: snow" }}

[x] Customize forwarded data

Data:

Copy

{
	"short_description": "{{ incident.title }}",
	"description": "This incident is created automatically by Grafana IRM:\n{{ incident.overviewURL }}",
	"state": "1",
	"u_irm_incident_id": "{{ incident_id }}",
	"caller_id": "grafana-irm"
}

Note the trigger template above. It’s there to avoid triggering the webhook when the incident being declared was created by the snow Service account, which we will set up in the next section. This is important because otherwise an incident created in ServiceNow would also trigger the webhook, and we don’t want to create a new incident there in that case.

We also set a special field—u_irm_incident_id (with the incident_id value)—in the data payload, so we keep a connection between the IRM incident and the ServiceNow incident (this will also be important in the next section).

Incident resolved

Name: Incident resolved

Trigger type: Incident Resolved

HTTP method: PUT

Webhook URL:

Copy

{%- for label in incident.labels -%}
{%- if label.key == "ServiceNow" -%}
https://devXXXXXX.service-now.com/api/now/table/incident/{{ label.label }}
{%- endif -%}
{%- endfor -%}

Username: grafana-irm

Password: [grafana-irm-snow-password]

[x] Customize forwarded data

Data:

Copy

{
"close_code": "Resolved by caller",
"close_notes": "{{ incident.summary }}",
"state": "6"
}

An important bit here is the URL template. It gets its value from an incident label that will contain the ServiceNow incident ID, which will be set by the ServiceNow business rule defined below.

From ServiceNow to IRM

Before defining a ServiceNow business rule to push updates to IRM, we will need a few things set up:

• In Grafana Cloud: Create a service account with the Admin role (name: snow; it should match the name in the trigger template filter in the previous section), and a corresponding service account token.
• In IRM: Confirm the ServiceNow label is available in IRM settings > Incident (the label key should match the label referenced in the URL template in the previous section; you can add the label with any random value here).
• In ServiceNow: Create a custom field in the incident table with the name u_irm_incident_id to store the IRM incident ID sent by the IRM webhook.

Then, we can create the ServiceNow business rule. For that, you can follow the instructions defined in the docs for the IRM-ServiceNow integration, but using the script found here instead (where you should replace AUTH_TOKEN value with your service account token and API_BASE_URL with your Grafana stack URL).

The script will send updates to the IRM Incident API to create, label, and update IRM incidents every time a ServiceNow incident changes, handling status updates and linking both incidents between them.

You can see the final outcome in the demo below:

Going further

Flexible webhook setup for incident-related events opens a wide set of options for integrations, allowing you to do things like announcing incidents in a different ChatOps service (e.g., Mattermost, Discord, etc), triggering an external workflow when a specific label is applied, or keeping a third-party service in sync on incident status changes.

To learn more, check out the Grafana Cloud IRM docs and start finding all the ways you can extend your incident response workflows by integrating IRM with your own services.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!