Test your IRM configuration

In this milestone, you test your configuration to verify that IRM creates an Alert Group from a firing alert and executes escalation and notification steps as expected.

To test your setup from an alert rule, complete the following steps:

1. Trigger an alert rule to fire.

   1. In the Alert rules UI, locate the alert rule connected to your IRM contact point.
   2. Open the rule and click Edit to lower the conditions to force the alert rule to fire (for example, CPU > 5%).
   3. If necessary, lower the evaulation period to 1 minute so it fires quickly.
   4. Save the rule and wait one evaluation cycle and confirm the rule shows a firing state.

2. Verify the new Alert group in IRM.

   1. In the IRM navigation menu, click Alert Groups.
   2. Locate the new Alert Group created by the firing rule.
   3. Click to open the Alert Group details.

3. In the timeline, verify escalation chain selection, the correct on-call user is identified, and note first notification attempts.

4. Verify escalation and notification steps are executed as expected.

   In the timeline, verify escalation chain selection, correct on-call user identified, and note first notification attempts. If you configured additional escalation steps, wait for the escalation delay and verify the next steps.

5. When you’re ready, acknowledge the Alert Group to cease all escalation and notification steps.

6. If necessary, restore the alert rule and confirm that it returns to a non-firing (OK) state after the next evaluation.

If the Alert Group doesn’t appear, confirm the alert rule includes the IRM contact point and actually fired. Check recent delivery attempts in Alerting → Contact points.

To test your setup from an IRM demo alert, complete the following steps:

1. In the IRM navigation menu, click Integrations and open your Grafana Alerting integration.
2. Click Send demo alert.
3. Review and optionally customize the demo alert payload.
4. Click Send Alert.
5. Verify escalation and notification steps are executed as expected.
6. When you’re ready, acknowledge and resolve the Alert Group.

Expected results

When your test is successful, you should observe the following:

• Alert Group creation: A new Alert Group appears in IRM → Alert Groups within a few seconds of the alert firing.
• Timeline activity: The Alert Group timeline shows:
  • Escalation chain selection with the correct chain name
  • On-call user identification based on your schedule
  • Notification attempts with timestamps for each configured notification method
  • Escalation steps progressing according to your configured delays
• Notification delivery: You receive notifications through your configured channels (email, mobile app, Slack, SMS, etc.) at the expected times.
• Acknowledgment behavior: When you acknowledge the Alert Group, the timeline shows acknowledgment and all further escalation and notifications stop.

Troubleshooting

If your test doesn’t produce the expected results, use the following troubleshooting steps:

If no Alert Group is created after triggering your alert:

1. Verify the alert rule is actually firing:
   • In Alerting → Alert rules, confirm the rule shows a Firing state.
   • Check that the rule evaluation period has elapsed since you made changes.
2. Confirm the IRM contact point is configured:
   • In the alert rule, verify the IRM contact point is listed in the Contact points section.
   • If missing, add it and save the rule.
3. Check alert delivery:
   • In Alerting → Contact points, open your IRM contact point and review Recent delivery attempts.
   • Look for any delivery errors or failures.

If the Alert Group appears but you don’t receive notifications:

1. Review notification rules:
   • In IRM → Settings → Personal notification rules, confirm you have active notification rules.
   • Check that your preferred notification channels (Phone, mobile app, etc.) are connected.
2. Check notification channel status:
   • For mobile push notifications, confirm the IRM mobile app is installed and logged in.
   • For SMS and phone calls, verify your phone number is verified in your notification preferences.

If the timeline doesn’t show expected escalation steps:

1. Verify escalation chain configuration:
   • In IRM → Escalation chains, confirm the chain attached to your integration includes the expected steps.
   • Check that escalation wait steps are configured correctly.
2. Confirm schedule attachment:
   • If your escalation chain uses a schedule, verify the schedule is attached to the correct step.
   • Check that the schedule has active on-call users for the current time period.
3. Review timeline details:
   • In the Alert Group timeline, look for any error messages or warnings about escalation chain execution.
   • Check if the escalation chain step shows “No one on-call” or similar status messages.

In the final milestone, you review what you accomplished and plan next steps.