Configure notification messages on Grafana Labs

Using Go's templating language

Tue, 21 May 2024 15:18:36 +0000

Using Go’s templating language

You write notification templates in Go’s templating language, text/template.

Before you start creating your own notification templates, we recommend that you read through this topic, which provides you with an overview of Go’s templating language and writing templates in text/template.

Dot

In text/template there is a special cursor called dot, and is written as .. You can think of this cursor as a variable whose value changes depending where in the template it is used. For example, at the start of a notification template . refers to something called ExtendedData which contains a number of fields including Alerts, Status, GroupLabels, CommonLabels, CommonAnnotations and ExternalURL. However, dot might refer to something else when used in a range over a list, when used inside a with, or when writing feature templates to be used in other templates. You can see examples of this in Create notification templates, and all data and functions in the Reference.

Opening and closing tags

In text/template, templates start with {{ and end with }} irrespective of whether the template prints a variable or executes control structures such as if statements. This is different from other templating languages such as Jinja where printing a variable uses {{ and }} and control structures use {% and %}.

Print

To print the value of something use {{ and }}. You can print the value of dot, a field of dot, the result of a function, and the value of a variable. For example, to print the Alerts field where dot refers to ExtendedData you would write the following:

{{ .Alerts }}

Iterate over alerts

To print just the labels of each alert, rather than all information about the alert, you can use a range to iterate the alerts in ExtendedData:

{{ range .Alerts }}
{{ .Labels }}
{{ end }}

Inside the range dot no longer refers to ExtendedData, but to an Alert. You can use {{ .Labels }} to print the labels of each alert. This works because {{ range .Alerts }} changes dot to refer to the current alert in the list of alerts. When the range is finished dot is reset to the value it had before the start of the range, which in this example is ExtendedData:

{{ range .Alerts }}
{{ .Labels }}
{{ end }}
{{/* does not work, .Labels does not exist here */}}
{{ .Labels }}
{{/* works, cursor was reset */}}
{{ .Status }}

Iterate over annotations and labels

Let’s write a template to print the labels of each alert in the format The name of the label is $name, and the value is $value, where $name and $value contain the name and value of each label.

Like in the previous example, use a range to iterate over the alerts in .Alerts such that dot refers to the current alert in the list of alerts, and then use a second range on the sorted labels so dot is updated a second time to refer to the current label. Inside the second range use .Name and .Value to print the name and value of each label:

{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
The name of the label is {{ .Name }}, and the value is {{ .Value }}
{{ end }}
{{ range .Annotations.SortedPairs }}
The name of the annotation is {{ .Name }}, and the value is {{ .Value }}
{{ end }}
{{ end }}

The index function

To print a specific annotation or label use the index function.

{{ range .Alerts }}
The name of the alert is {{ index .Labels "alertname" }}
{{ end }}

If statements

You can use if statements in templates. For example, to print There are no alerts if there are no alerts in .Alerts you would write the following:

{{ if .Alerts }}
There are alerts
{{ else }}
There are no alerts
{{ end }}

With

With is similar to if statements, however unlike if statements, with updates dot to refer to the value of the with:

{{ with .Alerts }}
There are {{ len . }} alert(s)
{{ else }}
There are no alerts
{{ end }}

Variables

Variables in text/template must be created within the template. For example, to create a variable called $variable with the current value of dot you would write the following:

{{ $variable := . }}

You can use $variable inside a range or with and it will refer to the value of dot at the time the variable was defined, not the current value of dot.

For example, you cannot write a template that use {{ .Labels }} in the second range because here dot refers to the current label, not the current alert:

{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
{{ .Name }} = {{ .Value }}
{{/* does not work because in the second range . is a label not an alert */}}
There are {{ len .Labels }}
{{ end }}
{{ end }}

You can fix this by defining a variable called $alert in the first range and before the second range:

{{ range .Alerts }}
{{ $alert := . }}
{{ range .Labels.SortedPairs }}
{{ .Name }} = {{ .Value }}
{{/* works because $alert refers to the value of dot inside the first range */}}
There are {{ len $alert.Labels }}
{{ end }}
{{ end }}

Range with index

You can get the index of each alert within a range by defining index and value variables at the start of the range:

{{ $num_alerts := len .Alerts }}
{{ range $index, $alert := .Alerts }}
This is alert {{ $index }} out of {{ $num_alerts }}
{{ end }}

Define templates

You can define templates using define and the name of the template in double quotes. You should not define templates with the same name as other templates, including default templates such as __subject, __text_values_list, __text_alert_list, default.title and default.message. Where a template has been created with the same name as a default template, or a template in another notification template, Grafana might use either template. Grafana does not prevent, or show an error message, when there are two or more templates with the same name.

{{ define "print_labels" }}
{{ end }}

Execute templates

You can execute defined templates using template, the name of the template in double quotes, and the cursor that should be passed to the template:

{{ template "print_labels" . }}

Pass data to templates

Within a template dot refers to the value that is passed to the template.

For example, if a template is passed a list of firing alerts then dot refers to that list of firing alerts:

{{ template "print_alerts" .Alerts }}

If the template is passed the sorted labels for an alert then dot refers to the list of sorted labels:

{{ template "print_labels" .SortedLabels }}

This is useful when writing reusable templates. For example, to print all alerts you might write the following:

{{ template "print_alerts" .Alerts }}

Then to print just the firing alerts you could write this:

{{ template "print_alerts" .Alerts.Firing }}

This works because both .Alerts and .Alerts.Firing is a list of alerts.

{{ define "print_alerts" }}
{{ range . }}
{{ template "print_labels" .SortedLabels }}
{{ end }}
{{ end }}

Comments

You can add comments with {{/* and */}}:

{{/* This is a comment */}}

To prevent comments from adding line breaks use:

{{- /* This is a comment with no leading or trailing line breaks */ -}}

Indentation

You can use indentation, both tabs and spaces, and line breaks, to make templates more readable:

{{ range .Alerts }}
  {{ range .Labels.SortedPairs }}
    {{ .Name }} = {{ .Value }}
  {{ end }}
{{ end }}

However, indentation in the template will also be present in the text. Next we will see how to remove it.

Remove spaces and line breaks

In text/template use {{- and -}} to remove leading and trailing spaces and line breaks.

For example, when using indentation and line breaks to make a template more readable:

{{ range .Alerts }}
  {{ range .Labels.SortedPairs }}
    {{ .Name }} = {{ .Value }}
  {{ end }}
{{ end }}

The indentation and line breaks will also be present in the text:

    alertname = "Test"

    grafana_folder = "Test alerts"

You can remove the indentation and line breaks from the text changing }} to -}} at the start of each range:

{{ range .Alerts -}}
  {{ range .Labels.SortedPairs -}}
    {{ .Name }} = {{ .Value }}
  {{ end }}
{{ end }}

The indentation and line breaks in the template are now absent from the text:

alertname = "Test"
grafana_folder = "Test alerts"

Create notification templates

Tue, 24 Sep 2024 07:09:39 +0000

Create notification templates

Create reusable notification templates to send to your contact points.

You can add one or more templates to your notification template.

Your notification template name must be unique. You cannot have two templates with the same name in the same notification template or in different notification templates. Avoid defining templates with the same name as default templates, such as: __subject, __text_values_list, __text_alert_list, default.title and default.message.

To create a notification template, complete the following steps.

Click Alerts & IRM -> Contact points.
Click the Notification Templates tab and then + Add notification template.
Enter a name for the notification template.
Write the content of the template in the content field.
Save your changes.

{{ define "email.subject" }} and {{ end }} is automatically added to the start and end of the content:

To create a notification template that contains more than one template:

Click + Add notification template.
Enter a name for the notification template.
Write each template in the Content field, including {{ define "name-of-template" }} and {{ end }} at the start and end of each template.
Save your changes.

Preview notification templates

Preview how your notification templates should look before using them in your contact points, helping you understand the result of the template you are creating as well as enabling you to fix any errors before saving it.

Note: This feature is only for Grafana Alertmanager.

To preview your notification templates:

Navigate to Alerts&IRM -> Alerting -> Contact points -> Notification Templates.
Click + Add notification template or edit an existing template.
Add or update your template content.

Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself or click Select alert instances or Add custom alerts.
Optional: To add alert data from existing alert instances:

a. Click Select alert instances.

b. Hover over the alert instances to view more information on each alert instance.

c. Click Confirm to add the alert instance(s) to the payload.
Optional: To add alert data using the Alert data editor, click Add custom data:

a. Add annotations, custom labels and/or set a dashboard or a panel.

b. Toggle Firing/resolved depending on whether you want to add firing or resolved alerts to your notification.

c. Click Add alert data.

d. Click Refresh preview to see what your template content should look like and the corresponding payload data.

If there are any errors in your template, they are displayed in the Preview and you can correct them before saving.
Save your changes.

Template the subject of an email

Template the subject of an email to contain the number of firing and resolved alerts:

1 firing alert(s), 0 resolved alerts(s)

Create a template called email.subject with the following content:

{{ define "email.subject" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}

Execute the template from the subject field in your contact point integration:
```
{{ template "email.subject" . }}
```

Template the message of an email

Template the message of an email to contain a summary of all firing and resolved alerts:

There are 2 firing alert(s), and 1 resolved alert(s)

Firing alerts:

- alertname=Test 1 grafana_folder=GrafanaCloud has value(s) B=1
- alertname=Test 2 grafana_folder=GrafanaCloud has value(s) B=2

Resolved alerts:

- alertname=Test 3 grafana_folder=GrafanaCloud has value(s) B=0

Create a notification template called email with two templates in the content: email.message_alert and email.message.

The email.message_alert template is used to print the labels and values for each firing and resolved alert while the email.message template contains the structure of the email.

{{- define "email.message_alert" -}}
{{- range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }} has value(s)
{{- range $k, $v := .Values }} {{ $k }}={{ $v }}{{ end }}
{{- end -}}

{{ define "email.message" }}
There are {{ len .Alerts.Firing }} firing alert(s), and {{ len .Alerts.Resolved }} resolved alert(s)

{{ if .Alerts.Firing -}}
Firing alerts:
{{- range .Alerts.Firing }}
- {{ template "email.message_alert" . }}
{{- end }}
{{- end }}

{{ if .Alerts.Resolved -}}
Resolved alerts:
{{- range .Alerts.Resolved }}
- {{ template "email.message_alert" . }}
{{- end }}
{{- end }}

{{ end }}

Execute the template from the message field in your contact point integration:
```
{{ template "email.message" . }}
```

Group multiple alert instances into one email notification

To make alerts more concise, you can group multiple instances of a firing alert into a single email notification in a table format. This way, you avoid long, repetitive emails and make alerts easier to digest.

Follow these steps to create a custom notification template that consolidates alert instances into a table.

Modify the alert rule to include an annotation that is referenced in the notification template later on.
Enter a name for the custom annotation: In this example, ServerInfo.
Enter the following code as the value for the annotation. It retrieves the server’s instance name and a corresponding metric value, formatted as a table row:
```
{{ index $labels "instance" }}{{- "\t" -}}{{ index $values "A"}}{{- "\n" -}}
```
This line of code returns the labels and their values in the form of a table. Assuming $labels has {"instance": "node1"} and $values has {"A": "123"}, the output would be:
```
node1    123
```
Create a notification template that references the ServerInfo annotation.
Go
```
{{ define "Table" }}
{{- "\nHost\t\tValue\n" -}}
{{ range .Alerts -}}
{{ range .Annotations.SortedPairs -}}
{{ if (eq .Name  "ServerInfo") -}}
{{ .Value -}}
{{- end }}
{{- end }}
{{- end }}
{{ end }}
```
The notification template outputs a list of server information from the “ServerInfo” annotation for each alert instance.
Navigate to your contact point in Grafana
In the Message field, reference the template by name (see Optional Email settings section):
```
{{ template "Table" . }}
```
This generates a neatly formatted table in the email, grouping information for all affected servers into a single notification.

Conditional notification template

Template alert notifications based on a label. In this example the label represents a namespace.

Use the following code in your notification template to display different messages based on the namespace:

{{ define "my_conditional_notification" }}
{{ if eq .CommonLabels.namespace "namespace-a" }}
Alert: CPU limits have reached 80% in namespace-a.
{{ else if eq .CommonLabels.namespace "namespace-b" }}
Alert: CPU limits have reached 80% in namespace-b.
{{ else if eq .CommonLabels.namespace "namespace-c" }}
Alert: CPU limits have reached 80% in namespace-c.
{{ else }}
Alert: CPU limits have reached 80% for {{ .CommonLabels.namespace }} namespace.
{{ end }}
{{ end }}

.CommonLabels is a map containing the labels that are common to all the alerts firing.

Make sure to replace the .namespace label with a label that exists in your alert rule.

Replace namespace-a, namespace-b, and namespace-c with your specific namespace values.
Navigate to your contact point in Grafana
In the Message field, reference the template by name (see Optional settings section):
```
{{ template "my_conditional_notification" . }}
```
This template alters the content of alert notifications depending on the namespace value.

Template the title of a Slack message

Template the title of a Slack message to contain the number of firing and resolved alerts:

1 firing alert(s), 0 resolved alerts(s)

Create a template called slack.title with the following content:

{{ define "slack.title" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}

Execute the template from the title field in your contact point integration:
```
{{ template "slack.title" . }}
```

Template the content of a Slack message

Template the content of a Slack message to contain a description of all firing and resolved alerts, including their labels, annotations, Silence URL and Dashboard URL.

Note:

This template is for Grafana-managed alerts only. To use the template for Grafana Mimir/Loki-managed alerts, delete the references to DashboardURL and SilenceURL. For more information, see the Prometheus documentation on notifications.

1 firing alert(s):

[firing] Test1
Labels:
- alertname: Test1
- grafana_folder: GrafanaCloud
Annotations:
- description: This is a test alert
Silence: https://example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest1&matcher=grafana_folder%3DGrafanaCloud
Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1

1 resolved alert(s):

[firing] Test2
Labels:
- alertname: Test2
- grafana_folder: GrafanaCloud
Annotations:
- description: This is another test alert
Silence: https://example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest2&matcher=grafana_folder%3DGrafanaCloud
Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1

Create a template called slack with two templates in the content: slack.print_alert and slack.message.

The slack.print_alert template is used to print the labels, annotations, SilenceURL and DashboardURL while the slack.message template contains the structure of the notification.

{{ define "slack.print_alert" -}}
[{{.Status}}] {{ .Labels.alertname }}
Labels:
{{ range .Labels.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end -}}
{{ if .Annotations -}}
Annotations:
{{ range .Annotations.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end -}}
{{ end -}}
{{ if .SilenceURL -}}
  Silence: {{ .SilenceURL }}
{{ end -}}
{{ if .DashboardURL -}}
  Go to dashboard: {{ .DashboardURL }}
{{- end }}
{{- end }}

{{ define "slack.message" -}}
{{ if .Alerts.Firing -}}
{{ len .Alerts.Firing }} firing alert(s):
{{ range .Alerts.Firing }}
{{ template "slack.print_alert" . }}
{{ end -}}
{{ end }}
{{ if .Alerts.Resolved -}}
{{ len .Alerts.Resolved }} resolved alert(s):
{{ range .Alerts.Resolved }}
{{ template "slack.print_alert" .}}
{{ end -}}
{{ end }}
{{- end }}

Execute the template from the text body field in your contact point integration:
```
{{ template "slack.message" . }}
```

Template both email and Slack with shared templates

Instead of creating separate notification templates for email and Slack, you can share the same template.

For example, if you want to send an email with this subject and Slack message with this title:

1 firing alert(s), 0 resolved alerts(s)

Create a template called common.subject_title with the following content:

{{ define "common.subject_title" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}

For email, execute the template from the subject field in your email contact point integration:
```
{{ template "common.subject_title" . }}
```
For Slack, execute the template from the title field in your Slack contact point integration:
```
{{ template "common.subject_title" . }}
```

Use notification templates

Tue, 21 May 2024 15:18:36 +0000

Use notification templates

Use templates in contact points to customize your notifications.

In the Contact points tab, you can see a list of your contact points.

To create a new contact point, click New.

Note: You can edit an existing contact by clicking the Edit icon.
Execute a template from one or more fields such as Message and Subject:

Use notification template

For more information on how to write and execute templates, refer to Using Go’s templating language and Create notification templates.
Click Save contact point.

Reference

Tue, 26 Mar 2024 12:30:09 +0000

Reference

Data

Alert

Name	Kind	Description	Example
Status	`string`	Firing or resolved	`{{ .Status }}`
Labels	`KV`	The labels for this alert	`{{ .Labels }}`
Annotations	`KV`	The annotations for this alert	`{{ .Annotations }}`
Values	`KV`	The values of all expressions, including Classic Conditions	`{{ .Values }}`
StartsAt	`Time`	The time the alert fired	`{{ .StartsAt }}`
EndsAt	`Time`		`{{ .EndsAt }}`
GeneratorURL	`string`	A link to Grafana, or the Alertmanager if using an external Alertmanager	`{{ .GeneratorURL }}`
SilenceURL	`string`	A link to silence the alert	`{{ .SilenceURL }}`
DashboardURL	`string`	A link to the Grafana Dashboard if the alert has a Dashboard UID annotation	`{{ .DashboardURL }}`
PanelURL	`string`	A link to the panel if the alert has a Panel ID annotation	`{{ .PanelURL }}`
Fingerprint	`string`	A unique string that identifies the alert	`{{ .Fingerprint }}`
ValueString	`string`	A string that contains the labels and value of each reduced expression in the alert.	`{{ .ValueString }}`

ExtendedData

Name	Kind	Description	Example
Receiver	`string`	The name of the contact point sending the notification	`{{ .Receiver }}`
Status	`string`	The status is `firing` if at least one alert is firing, otherwise `resolved`	`{{ .Status }}`
Alerts	`[]Alert`	List of all firing and resolved alerts in this notification	`There are {{ len .Alerts }} alerts`
Firing alerts	`[]Alert`	List of all firing alerts in this notification	`There are {{ len .Alerts.Firing }} firing alerts`
Resolved alerts	`[]Alert`	List of all resolved alerts in this notification	`There are {{ len .Alerts.Resolved }} resolved alerts`
GroupLabels	`KV`	The labels that group these alerts in this	`{{ .GroupLabels }}`
CommonLabels	`KV`	The labels common to all alerts in this notification	`{{ .CommonLabels }}`
CommonAnnotations	`KV`	The annotations common to all alerts in this notification	`{{ .CommonAnnotations }}`
ExternalURL	`string`	A link to Grafana, or the Alertmanager that sent this notification if using an external Alertmanager	`{{ .ExternalURL }}`

KV

KV is a set of key value pairs, where each key and value is a string. If a KV happens to contain numbers or bools then these are string representations of the numeric or boolean value.

Here is an example of a KV, the annotations of an alert:

summary: 'A summary of the alert'
description: 'A description of the alert'

In addition to iterating over each key value pair, you can sort the pairs, remove keys, and iterate over just the keys or the values.

Name	Description	Arguments	Example
SortedPairs	Sorts		`{{ .Annotations.SortedPairs }}`
Remove	Returns a copy of the KV with the keys removed	[]string	`{{ .Annotations.Remove "summary" }}`
Names	A list of the names		`{{ .Names }}`
Values	A list of the values		`{{ .Values }}`

Time

Time is from the Go time package. You can print a time in a number of different formats. For example, to print the time that an alert fired in the format Monday, 1st January 2022 at 10:00AM you would write the following template:

{{ .StartsAt.Format "Monday, 2 January 2006 at 3:04PM" }}

You can find a reference for Go’s time format here.

Use images in notifications

Tue, 26 Mar 2024 12:30:09 +0000

Use images in notifications

Images in notifications helps recipients of alert notifications better understand why an alert has fired or resolved by including a screenshot of the panel associated with the alert.

Note
This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager

When an alert is fired or resolved Grafana takes a screenshot of the panel associated with the alert. This is determined via the Dashboard UID and Panel ID annotations of the rule. Grafana cannot take a screenshot for alerts that are not associated with a panel.

Grafana takes at most two screenshots for each alert: once when the alert fires and again when the alert is resolved. Screenshots are not re-taken over the lifetime of the alert, instead you should open the panel in Grafana to follow the data in real time. In addition, depending on how alerts are grouped in your notification policies, Grafana might send a notification with many screenshots of the same panel. This happens because Grafana does not know how your alerts are grouped at the time a screenshot is taken, and so acts conservatively by taking a screenshot for every alert.

Once a screenshot has been taken Grafana can either upload it to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; upload the screenshot to it’s internal web server; or upload it to the service that is receiving the notification, such as Slack. Which option you should choose depends on how your Grafana is managed and which integrations you use. More information on this can be found in Requirements.

Refer to the table at the end of this page for a list of contact points and their support for images in notifications.

Requirements

To use images in notifications, Grafana must be set up to use image rendering. You can either install the image rendering plugin or run it as a remote rendering service.
When a screenshot is taken it is saved to the [data][paths] folder, even if Grafana is configured to upload screenshots to a cloud storage service. Grafana must have write-access to this folder otherwise screenshots cannot be saved to disk and an error will be logged for each failed screenshot attempt.
You should use a cloud storage service unless sending alerts to Discord, Email, Pushover, Slack or Telegram. These integrations support either embedding screenshots in the email or attaching screenshots to the notification, while other integrations must link screenshots uploaded to a cloud storage bucket. If a cloud storage service has been configured then integrations that support both will link screenshots from the cloud storage bucket instead of embedding or attaching screenshots to the notification.
If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; then image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens we recommend using integrations which support uploading images or disabling images in notifications altogether.
When uploading screenshots to a cloud storage service Grafana uses a random 20 character (30 characters for Azure Blob Storage) filename for each image. This makes URLs hard to guess but not impossible.
Grafana does not delete screenshots from cloud storage. We recommend configuring a retention policy with your cloud storage service to delete screenshots older than 1 month.
If Grafana is configured to upload screenshots to its internal web server, and accessing Grafana requires logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens we recommend using integrations which support uploading images or disabling images in notifications altogether.
Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from static_root_path/images/attachments after a certain amount of time we recommend setting up a CRON job.

Configuration

Note
Grafana Cloud users can request this feature by opening a support ticket in the Cloud Portal.

Having installed either the image rendering plugin, or set up Grafana to use a remote rendering service, set capture in [unified_alerting.screenshots] to true:

# Enable screenshots in notifications. You must have either installed the Grafana image rendering
# plugin, or set up Grafana to use a remote rendering service.
# For more information on configuration options, refer to [rendering].
capture = false

If screenshots should be uploaded to cloud storage then upload_external_image_storage should also be set to true:

# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false

Restart Grafana for the changes to take effect.

Advanced configuration

We recommended that max_concurrent_screenshots is less than or equal to concurrent_render_request_limit. The default value for both max_concurrent_screenshots and concurrent_render_request_limit is 5:

# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5

Supported contact points

Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading screenshots to the receiving service and referencing screenshots that have been uploaded to a cloud storage service.

Name	Upload from disk	Reference from cloud storage
DingDing	No	No
Discord	Yes (Maximum of 10 per notification)	Yes (Maximum of 10 per notification)
Email	Yes (Embedded in the email)	Yes
Google Chat	No	Yes
Kafka	No	No
Line	No	No
Microsoft Teams	No	Yes
Opsgenie	No	Yes
Pagerduty	No	Yes
Prometheus Alertmanager	No	No
Pushover	Yes (Maximum of 1 per notification)	No
Sensu Go	No	No
Slack	Yes (when using Bot tokens, maximum of 5 per notification)	Yes (when using webhooks, maximum of 1 per notification)
Telegram	Yes	No
Threema	No	No
VictorOps	No	No
Webhook	No	Yes

Limitations

This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
A number of contact points support at most one image per notification. In this case, just the first image is either uploaded to the receiving service or referenced from cloud storage per notification.
When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.
If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN.

Troubleshooting

If Grafana has been set up to send images in notifications, however notifications are still being received without them, follow the troubleshooting steps below:

Check that images in notifications has been set up as per the instructions.
Enable debug logging in Grafana and look for logs with the logger ngalert.image.
If the alert is not associated with a dashboard there will be logs for Cannot take screenshot for alert rule as it is not associated with a dashboard.
If the alert is associated with a dashboard, but no panel in the dashboard, there will be logs for Cannot take screenshot for alert rule as it is not associated with a panel.
If images cannot be taken because of mis-configuration or an issue with image rendering there will be logs for Failed to take an image including the Dashboard UID, Panel ID, and the error message.
Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service.

Metrics

Grafana provides the following metrics to observe the performance and failure rate of images in notifications. For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter grafana_screenshot_failures_total is updated.

grafana_alerting_image_cache_hits_total
grafana_alerting_image_cache_misses_total
grafana_screenshot_duration_seconds
grafana_screenshot_failures_total
grafana_screenshot_successes_total
grafana_screenshot_upload_failures_total
grafana_screenshot_upload_successes_total