Documentation Grafana Cloud Alerts and IRM IRM Manage Insights and reporting Incident insights

Incident insights

Incident insights in Grafana IRM enables you to visualize, analyze, and track incident response data. Use these insights to measure your organization’s response effectiveness, identify trends, and continuously improve your incident management processes.

About incident insights

Incident insights provide:

• A pre-built dashboard with key incident response metrics
• The ability to build custom dashboards using the Grafana Incident data source
• Flexible filtering and exploration by labels, severity, and other dimensions
• Customizable dashboard templates to fit your organization’s needs
• Embedding of incident metrics into existing Grafana dashboards

Incident insights are included with all Grafana IRM plans at no additional cost.

Set up incident insights

The incident insights dashboard is created automatically when you enable the feature.

To set up and access incident insights for the first time:

1. In the Grafana Cloud main menu, navigate to Alerts & IRM > IRM.
2. Select Insights, then click the Incidents tab.
3. Click Set up Insights dashboard.
4. You are redirected to the insights dashboard upon completion.

During setup, Grafana IRM:

• Configures the Grafana Incident data source
• Generates a service account and token with viewer permissions
• Creates the pre-built Insights dashboard

To revisit the dashboard later, select Insights from the left menu.

Understand your incident metrics

Grafana IRM tracks several core metrics to help you understand and improve your incident response:

• Mean Time To Resolution (MTTR):
  The average time from when an incident starts to when it is resolved.
  Formula: incidentEnd - incidentStart

  • If incidentEnd is not set manually, it defaults to the time the incident was marked as resolved (closedTime).

• Mean Time To Detect (MTTD):
  The average time from when an incident starts to when it is declared.
  Formula: incidentCreated - incidentStart

Incident field definitions:

Query syntax

The Incident data source uses the incident query syntax, which is also used to filter the incident in Grafana IRM You can reuse any bookmarked or saved expressions from Grafana IRM.

Example queries

• Critical or security-related incidents:

  Copy

  or(severity:critical label:security)

• Active incidents within a timeframe:

  Copy

  status:active started:${__from:date}, ${__to:date}

• Resolved incidents without a specific label:

  Copy

  status:resolved -label:value

• Resolved incidents with multiple specific labels:

  Copy

  status:resolved or(label:'squad:incident' label:PIR)

Filter incidents

When filtering by multiple values:

• Label filters (e.g., service_name:accounts service_name:orders) use AND logic. Only incidents with all specified label values will match.
• Other filters (like status:active status:resolved) use OR logic.

Correct or modify incident timelines

If your metrics are inaccurate due to late declaration or resolution of an incident:

• Adjust incidentStart or incidentEnd using the timeline editor.
• incidentCreated cannot be changed. If this affects your MTTD, consider annotating the incident or filtering it out in your analysis.

Create custom insights dashboards

You can create custom dashboards to visualize incident data that matters most to your teams:

1. In Grafana Cloud, go to Dashboards > New Dashboard.
2. Add a new panel.
3. Select the Grafana Incident data source.
4. Build queries using the syntax described in the following sections.
5. Choose visualizations to highlight key metrics.

Use variables in queries

The Query Editor supports Grafana Variables, which allows you to create interactive dashboards. Variables allow you to create dynamic dashboards where users can select different values to update the visualizations.

You can create variables for:

• Status: Filter by incident status (active, resolved, etc.)
• Severity: Filter by incident severity levels
• Labels: Filter by incident labels

Example: Creating a variable for incident labels

1. Create a new dashboard variable.
2. Select the Grafana Incident data source.
3. Use a query that returns label values.
4. Configure the variable to allow multi-selection.

Example query using variables:

Copy

status:$Status severity:$Severities label:$Labels

Troubleshoot dashboard variables or queries

If your dashboard variables or queries are not working, you may be using an outdated Insights dashboard template. To update, consult the latest dashboard template examples or rebuild your dashboard using the current variable names and query syntax.

Analyze incident data

Key metrics for incident analysis include:

• Mean Time To Resolution (MTTR): Average time from incident start to resolution
• Mean Time To Detect (MTTD): Average time from incident start to declaration
• Incident frequency: Number of incidents over time, by severity or service
• Team impact: Distribution of incidents across teams and services
• Root cause patterns: Common themes in incident causes identified through labels
• Time of day patterns: When incidents are most likely to occur

Best practices

• Start with the pre-built dashboard to understand available metrics
• Create custom dashboards for specific teams or services
• Use consistent labeling to enable meaningful filtering and grouping
• Track trends over time rather than focusing on point-in-time metrics
• Compare related metrics such as MTTR alongside incident count

Next steps

• Learn about alert groups insights to complement your incident metrics
• Configure incident management to improve your processes
• Set up dashboards for custom visualizations