Configure Grafana Cloud Asserts notifications
Grafana Cloud

Configure Grafana Cloud Asserts notifications

This topic describes how to configure Asserts notifications so that you can link back to Asserts RCA workbench from SLO and Asserts notification alerts.

For more information about how Asserts notification alerts work, refer to How notifications work in Grafana Cloud Asserts.

Note

If you use the mimirtool for syncing rule group state, this can cause unauthorized errors because Asserts was enabled. Errors occur because Asserts rules are stored in a protected namespace that our users can’t change.

Before you instruct mimirtool to ignore the Asserts namespace, disable Asserts. After you have set up ignoring the namespace with mimirtool, re-enable Asserts.

Before you begin

Before you configure Grafana Cloud Asserts notifications, ensure that you have configured a contact point.

Note

Asserts notifications are only supported with the Prometheus Alertmanager and not the Grafana Alertmanager.

Configure SLO alerts

If you are managing SLOs from Asserts, you must create a notification policy and configure an Alertmanager template.

Create a notification policy

To create a notification policy, complete the following steps:

  1. In the left-side menu, click Alerts & IRM > Alerting > Notification policies.

  2. From the Choose Alertmanager drop-down menu, select the Prometheus Alertmanager.

    By default, the Grafana Alertmanager is selected.

  3. Click +New child policy from the default policy.

  4. In the Matching labels section, add the following labels and values:

    LabelOperatorValue
    grafana_slo_provenance=asserts
    grafana_slo_severity=critical
  5. In the Contact point drop-down menu, select the contact point to send notifications to.

  6. Select the Override grouping toggle.

  7. Select or enter the following Group by values:

    • alertname
    • asserts_env
    • namespace
  8. Click Save policy.

Configure Alertmanager template

If Asserts manages your SLO, you can link back to RCA workbench by modifying an existing Alertmanager template.

Modify Alertmanager template

You can modify your existing Alertmanager template to include a URL that links back to Asserts RCA workbench.

The following code snippet shows how to modify a Slack template:

{{- if eq .CommonLabels.grafana_slo_provenance "asserts" }}
  :fire: *<INSTANCE_URL/a/grafana-asserts-app/assertions?env[0]={{ .CommonLabels.asserts_env }}{{ with .CommonLabels.asserts_site }}&site[0]={{ . }}{{ end }}&grafana_slo_uuid={{ .CommonLabels.grafana_slo_uuid }}&start={{(index .Alerts 0).StartsAt.Unix}}000-30m&end={{(index .Alerts 0).StartsAt.Unix}}000%2B30m|Asserts RCA Workbench>*
{{- end }}

Where INSTANCE_URL is the URL of the instance Asserts is running, for example:

https://acme.grafana.net

Configure Asserts notification alerts

Asserts also lets you to create notifications from assertions. For more information about how Asserts notification alerts work, refer to How notifications work in Grafana Cloud Asserts.

To configure an Asserts notification alert, you must create a notification policy and configure an Alertmanager template.

Create a notification policy

To create a notification policy, complete the following steps:

  1. In the left-side menu, click Alerts & IRM > Alerting > Notification policies.

  2. From the Choose Alertmanager drop-down menu, select the Prometheus Alertmanager.

    By default, the Grafana Alertmanager is selected.

  3. Click +New child policy from the default policy.

  4. In the Matching labels section, add the following labels and values:

    LabelOperatorValue
    asserts_alert_category=~.+
    alertname=NOTIFICATION_ALERTS
  5. In the Contact point drop-down, select the contact point to send notifications to.

  6. Select the Override grouping toggle.

  7. Select or enter the following Group by values:

    • alertname
    • asserts_env
    • asserts_notification_rule_name
    • namespace
    • source_alertname
  8. Click Save policy.

Configure Alertmanager template

When using Asserts notifications, you can link back to RCA workbench by modifying an existing Alertmanager template.

Modify Alertmanager template

You can modify your existing Alertmanager template to include a URL that links back to Asserts RCA workbench.

The following code snippet shows how to modify a Slack template:

{{- if eq .GroupLabels.alertname "NOTIFICATION_ALERTS" }}
  {{- $url := "INSTANCE_URL/a/grafana-asserts-app/alert-affected-entity?" }}
  {{- range $key, $value := (index .Alerts 0).Labels }}
    {{- $url = printf "%s%s=%s&" $url ($key | urlquery) ($value | urlquery) }}
  {{- end }}
  {{- $start := (index .Alerts 0).StartsAt.Unix}}
  {{- $url = printf "%salert_start_time=%d000-30m&alert_end_time=%d000%%2B30m" $url $start $start }}
  :fire: *<{{ $url }}|Asserts RCA Workbench>*
{{- end }}

Where INSTANCE_URL is the URL of the instance Asserts is running, for example:

https://acme.grafana.net