Documentation Grafana Cloud Alerts and IRM IRM Guides Best practices Alert groups

Best practices for alert groups

Alert grouping determines how IRM combines individual alerts into manageable units. Effective grouping reduces noise and ensures responders see related alerts together.

Understand alert grouping

Alert grouping is one of the most impactful configurations in IRM. The difference between good and poor grouping can mean receiving 1 phone call instead of 10 at 4am.

What is an alert group

An alert group is a collection of related alerts that IRM treats as a single unit. When alerts group together:

• Responders receive one notification instead of many.
• Related alerts appear together for easier triage.
• Actions like acknowledge or resolve apply to all alerts in the group.

How grouping relates to routing

Routing and grouping work together but happen in a specific order:

1. Routing: IRM evaluates routes to determine which escalation chain handles the alert.
2. Grouping: IRM evaluates the grouping template to determine which alert group receives the alert.

This order has important implications:

• Alerts on different routes never group together, even with the same grouping ID.
• Each alert payload is processed independently.

For more information about routing, refer to Alert routing best practices.

How grouping works

IRM uses a Jinja template to generate a grouping ID for each alert. Alerts with the same ID, on the same route, group together.

The grouping process

1. An alert arrives at an IRM integration.
2. IRM evaluates routes and selects the first match.
3. IRM evaluates the grouping template against the alert payload to produce an ID.
4. The alert joins an existing open group with the same ID, or creates a new group.

Open vs resolved groups

An open alert group is one in a Firing, Acknowledged, or Silenced state. A resolved alert group is closed to new alerts by default.

When a new alert arrives:

• If an open group exists with the same ID, the alert joins that group.
• If no open group exists, IRM creates a new group.

Configure grouping templates

Grouping templates control how IRM combines alerts. The right configuration depends on your alert source.

Alertmanager sources

For Grafana Alerting and Prometheus Alertmanager, keep the default template:

jinja2 Copy

{{ payload.groupKey }}

Why this matters:

• IRM trusts the groupKey from the payload, which encodes how Alertmanager grouped the alerts.
• Changing the template breaks auto-resolution because the payload status refers to the original grouping.
• IRM doesn’t parse individual alerts in the payload, only the top-level fields.

Configure grouping at the source:

For Grafana Alerting, use Notification Policies to configure grouping. Common grouping strategies include:

• Default: alertname + grafana_folder
• More granular: alertname + namespace + cluster

Other integrations

For webhooks and non-Alertmanager integrations, create custom grouping templates:

jinja2 Copy

{{ payload.commonLabels.service_name }}:{{ payload.commonLabels.namespace }}:{{ payload.commonLabels.cluster }}

This groups alerts by service, namespace, and cluster.

Grouping strategies

Choose a strategy based on how you want to organize alerts:

Choose good identifiers

Group by values that remain consistent across related alerts.

Good identifiers:

• alertname
• service_name
• namespace
• groupKey (for Alertmanager sources)

Avoid:

• Timestamps that change with each alert.
• Unique event IDs that differ per instance.
• Status fields like Firing or Resolved.

Test grouping templates

Before deploying changes to production:

1. Use the template preview feature in the integration settings.
2. Test with real alert payloads from your monitoring system.
3. Verify alerts group as expected.

Alert group lifecycle

After alerts are grouped, IRM manages their lifecycle through resolution and silencing.

Auto-resolution

Enable Autoresolve (called allow_source_based_resolving in the API/Terraform) for integrations that send resolve signals.

When enabled:

• IRM matches resolve signals to the corresponding alert group.
• The alert group resolves automatically when the source sends a resolve signal.

When disabled:

• IRM creates new, already-resolved alert groups for resolve signals.
• This adds clutter without providing value.

Best practice: Enable auto-resolution for Alertmanager-style integrations where alerts fire and resolve over time.

Silencing behavior

Silencing suppresses notifications but keeps the alert group open.

Time-limited silencing:

• The alert group reopens after the silence period ends.
• Use for temporary suppression during known issues.

Silence forever:

• The alert group remains open indefinitely.
• The group continues to receive matching alerts without notification.
• This can accumulate large numbers of alerts over time.

Best practice: Use time-limited silencing when possible. If you use “Silence forever,” periodically review silenced groups.

Resolution notes

Resolution notes capture context about how you resolved an alert group. They help build institutional knowledge and improve post-incident reviews.

To require resolution notes, enable Require resolution note (called is_resolution_note_required in the API/Terraform) in organization settings.

When enabled, users must add a resolution note before resolving an alert group.

Common pitfalls

Avoid these mistakes when configuring alert grouping.

Never group by status

Don’t include alert status (Firing, Resolved, OK) in grouping templates.

Problem: Firing and resolved alerts for the same issue end up in different groups. This breaks auto-resolution and defeats the purpose of grouping.

Don’t do this:

jinja2 Copy

{{ payload.status }}:{{ payload.commonLabels.alertname }}

Changing Alertmanager grouping templates

Don’t modify the default {{ payload.groupKey }} template for Alertmanager sources.

Problem: The payload’s status field refers to the original grouping. Changing the template causes mismatches between what Alertmanager grouped and what IRM receives.

Overusing silence forever

Don’t use “Silence forever” without a plan to review.

Problem: Silenced groups stay open and accumulate alerts indefinitely. Without periodic review, you may miss important signals hidden in silenced groups.

Best practices summary

• Keep Alertmanager defaults: Configure grouping at the source, not in IRM.
• Use stable identifiers: Group by values that don’t change per alert instance.
• Never group by status: Avoid Firing/Resolved in templates.
• Enable auto-resolution: For integrations that send resolve signals.
• Test before deploying: Preview templates with real payloads.
• Use time-limited silencing: Review any “Silence forever” groups periodically.
• Require resolution notes: Capture knowledge when resolving alerts.

Next steps

• Alert routing best practices
• Configure alert routing in detail
• Create alert templates for grouping and labeling
• Escalation chains best practices