Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Alerts and IRMbreadcrumb arrow IRMbreadcrumb arrow Guidesbreadcrumb arrow Best practicesbreadcrumb arrow Alert routing
Grafana Cloud

Best practices for alert routing

Alert routing in Grafana IRM determines which escalation chain handles each alert. Effective routing ensures alerts reach the right team quickly without manual intervention.

Key insight: Routing happens before grouping. Alerts on different routes never group together, even if they would produce the same grouping ID.

Two stages of routing

Alerts typically pass through two routing stages before reaching responders.

Stage 1: Source routing

Your alerting tool routes alerts to the appropriate IRM integration.

Grafana Alerting:

  • Notification Policies route alerts to Contact Points based on labels.
  • Use direct routing or hierarchical policies.

Prometheus Alertmanager:

  • Routing rules match alerts to receivers.
  • Supports label-based routing with regular expressions.

Other tools:

  • Each tool has its own routing mechanism.
  • Configure your tool to send alerts to the appropriate IRM webhook.

Use source routing for coarse-grained routing at the team or domain level.

Stage 2: IRM routing

Routes in IRM match alerts to escalation chains within an integration. Use IRM routing for fine-grained routing at the severity or service level.

  1. Use your alerting tool to route alerts to team-specific IRM integrations.
  2. Use IRM routes within each integration for severity-based escalation.

This separation keeps routing logic organized and maintainable.

Route configuration

Routes determine which escalation chain handles an alert group. IRM evaluates routes in order, and the first match wins.

Since routing happens before grouping, routes also determine which alerts can potentially group together. Alerts on different routes are isolated and never group with each other.

Route ordering

Order routes from most specific to least specific:

  1. Specific routes first: Match exact label values or complex conditions.
  2. Broader routes next: Match team or domain level.
  3. Default route last: Catches unmatched alerts.

Example order:

text 
1. severity="critical" AND service="payments" → payments-critical-chain
2. severity="critical" → critical-chain
3. team="platform" → platform-chain
4. (default) → default-chain

Route matching options

Routes can match using:

  • Labels: Match on extracted alert group labels. Clearest and easiest to audit.
  • Jinja2 expressions: Complex boolean logic on payload data. Provides flexibility for complex conditions.
  • Regex: Pattern matching on alert group title or payload fields. This is deprecated and no longer available in new integrations.

Choose the matching method that best fits your routing needs.

Default route best practices

The default route catches all alerts that don’t match other routes. Don’t treat it as a place for alerts you want to ignore.

Why this matters:

  • New or misconfigured alerts fall through to the default route.
  • If the default route is ignored, you may miss critical alerts.
  • Alerts you haven’t accounted for could go unnoticed.

Recommended approach:

  • Route the default to a real escalation chain, even a low-priority one.
  • Consider making the default route your critical path.
  • Add specific routes to filter out known low-priority alerts before the default.

This ensures unexpected alerts are visible rather than silently ignored.

Routing and grouping interaction

Understanding how routing and grouping interact is critical for effective IRM configuration.

Core principle: Routing always happens before grouping.

When an alert arrives at an IRM integration:

  1. Routing: IRM evaluates routes to determine which escalation chain handles the alert.
  2. Grouping: IRM evaluates the grouping template to determine which alert group receives the alert.

This order has important implications:

  • Routes are isolated: Alerts on different routes never group together, even if they would produce the same grouping ID.
  • Each payload is independent: IRM processes each alert payload as an immutable, independent event.
  • No cross-route grouping: An alert routed to “payments-critical” and an alert routed to “payments-warning” remain in separate alert groups.

Audit your routing

Regularly check for alerts going to default routes.

Review default route alerts

  1. Check the default route in your IRM integration.
  2. Review recent alert groups that matched the default route.
  3. Identify which labels are missing and update the alert rules.

Common causes

Alerts reaching the default route often indicate:

  • Missing routing labels on the alert.
  • Route conditions that don’t match the alert.
  • New alert types not yet configured.

Monitor routing effectiveness

Track the percentage of alerts that match specific routes versus the default route. Set a target, for example 95%, and work to improve routing coverage over time.

Best practices summary

  • Use two-stage routing: Coarse-grained at source, fine-grained in IRM.
  • Order routes correctly: Most specific first, default last.
  • Don’t ignore the default route: Ensure unexpected alerts are visible.
  • Use labels for routing: Clearer and easier to audit than regex.
  • Understand routing isolation: Alerts on different routes never group together.
  • Audit regularly: Check alerts going to default routes.

Next steps