Menu
Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Knowledge graphbreadcrumb arrow Configurebreadcrumb arrow Notifications
Grafana Cloud

Configure Grafana Cloud Knowledge Graph notifications

This topic describes how to configure knowledge graph notifications so that you can link back to RCA workbench from SLO and knowledge graph notification alerts.

For more information about how knowledge graph notification alerts work, refer to How notifications work in Grafana Cloud Knowledge Graph.

Note

If you use the mimirtool for syncing rule group state, this can cause unauthorized errors because the knowledge graph was enabled. Errors occur because the knowledge graph rules are stored in a protected namespace that our users can’t change.

Before you instruct mimirtool to ignore the knowledge graph namespace, disable the knowledge graph. After you have set up ignoring the namespace with mimirtool, re-enable the knowledge graph.

Before you begin

Before you configure Grafana Cloud Knowledge Graph notifications, ensure that you have configured a contact point.

Note

Knowledge graph notifications are only supported with the Prometheus Alertmanager and not the Grafana Alertmanager.

Configure SLO alerts

If you are managing SLOs from the knowledge graph, you must create a notification policy and configure an Alertmanager template.

Create a notification policy

To create a notification policy, complete the following steps:

  1. In the left-side menu, click Alerts & IRM > Alerting > Notification policies.

  2. From the Choose Alertmanager drop-down menu, select the Prometheus Alertmanager.

    By default, the Grafana Alertmanager is selected.

  3. Click +New child policy from the default policy.

  4. In the Matching labels section, add the following labels and values:

    LabelOperatorValue
    grafana_slo_provenance=asserts
    grafana_slo_severity=critical

  5. In the Contact point drop-down menu, select the contact point to send notifications to.

  6. Select the Override grouping toggle.

  7. Select or enter the following Group by values:

    • alertname
    • asserts_env
    • namespace

  8. Click Save policy.

Configure Alertmanager template

If the knowledge graph manages your SLO, you can link back to RCA workbench by modifying an existing Alertmanager template.

Modify Alertmanager template

You can modify your existing Alertmanager template to include a URL that links back to RCA workbench.

The following code snippet shows how to modify a Slack template:

{{- if eq .CommonLabels.grafana_slo_provenance "asserts" }}
  :fire: *<INSTANCE_URL/a/grafana-asserts-app/assertions?env[0]={{ .CommonLabels.asserts_env }}{{ with .CommonLabels.asserts_site }}&site[0]={{ . }}{{ end }}&grafana_slo_uuid={{ .CommonLabels.grafana_slo_uuid }}&start={{(index .Alerts 0).StartsAt.Unix}}000-30m&end={{(index .Alerts 0).StartsAt.Unix}}000%2B30m|Asserts RCA workbench>*
{{- end }}

Where INSTANCE_URL is the URL of the instance the knowledge graph is running, for example:

https://acme.grafana.net

Configure notification alerts

The knowledge graph also lets you to create notifications from assertions. For more information about how notification alerts work, refer to How notifications work in Grafana Cloud Knowledge Graph.

To configure a knowledge graph notification alert, you must create a notification policy and configure an Alertmanager template.

Create a notification policy

To create a notification policy, complete the following steps:

  1. In the left-side menu, click Alerts & IRM > Alerting > Notification policies.

  2. From the Choose Alertmanager drop-down menu, select the Prometheus Alertmanager.

    By default, the Grafana Alertmanager is selected.

  3. Click +New child policy from the default policy.

  4. In the Matching labels section, add the following labels and values:

    LabelOperatorValue
    asserts_alert_category=~.+
    alertname=NOTIFICATION_ALERTS

  5. In the Contact point drop-down, select the contact point to send notifications to.

  6. Select the Override grouping toggle.

  7. Select or enter the following Group by values:

    • alertname
    • asserts_env
    • asserts_notification_rule_name
    • namespace
    • source_alertname

  8. Click Save policy.

Note

For notification alerts, the alertname label is always NOTIFICATION_ALERTS. To get the original/actual alertname, use source_alertname.

Configure Alertmanager template

When using knowledge graph notifications, you can link back to RCA workbench by modifying an existing Alertmanager template.

Modify Alertmanager template

You can modify your existing Alertmanager template to include a URL that links back to RCA workbench.

The following code snippet shows how to modify a Slack template:

{{- if eq .GroupLabels.alertname "NOTIFICATION_ALERTS" }}
  {{- $url := "INSTANCE_URL/a/grafana-asserts-app/alert-affected-entity?" }}
  {{- range $key, $value := (index .Alerts 0).Labels }}
    {{- $url = printf "%s%s=%s&" $url ($key | urlquery) ($value | urlquery) }}
  {{- end }}
  {{- $start := (index .Alerts 0).StartsAt.Unix}}
  {{- $url = printf "%salert_start_time=%d000-30m&alert_end_time=%d000%%2B30m" $url $start $start }}
  :fire: *<{{ $url }}|Asserts RCA Workbench>*
{{- end }}

Where INSTANCE_URL is the URL of the instance the knowledge graph is running, for example:

https://acme.grafana.net