Manage and investigate alerts
Grafana Assistant interacts with Grafana Alerting to help you find firing alerts, understand their history, and manage silences without leaving the chat. You can also inspect routing policies and add context to alert rules to speed up future investigations.
What you’ll achieve
- Find and inspect alert rules instantly by name or status.
- Analyze alert history to see when alerts fired or resolved.
- Manage silences to reduce noise during maintenance.
- Audit routing configuration to ensure the right teams are notified.
- Enrich alert rules with automated Assistant context.
Before you begin
- Grafana Alerting must be enabled.
- Grafana-managed alert rules only: These tools work with Grafana-managed alert rules. Data source-managed alert rules (e.g., Mimir, Loki) are not supported.
- To list and inspect rules and policies, your user account requires Viewer permissions or higher.
- To create silences or modify rules, you must have Editor or Admin permissions.
Explore and inspect alerts
Quickly find what’s firing or look up specific rules to understand their configuration.
List and filter alerts
Ask the Assistant to show you alerts based on their state or name.
Show me all firing alerts Show me firing alerts in the “Production” folder Find alert rules related to “cpu usage”
Inspect rule details
Get a comprehensive view of a rule’s configuration, including its query, threshold, and current health.
Show details for the “High CPU Usage” alert rule What is the configuration of the “Kubelet down” alert?
View version history
Track changes to alert rules over time to correlate modifications with incidents.
Show me the version history for the “High CPU Usage” rule Who last modified the “Disk Space Low” alert?
Investigate alert history
Understand the behavior of your alerts over time. The Assistant can query the alert historian to show state transitions and notifications.
Analyze state transitions
See exactly when an alert moved from Normal to Pending or Firing.
When did “High CPU Usage” last fire? Show me the alert state history for “High CPU Usage” for the last 24 hours
Check notifications
Verify if and when notifications were sent for a specific alert.
Show me the notification history for the “High CPU Usage” alert Did the “Disk Full” alert send a notification yesterday?
Manage silences
Reduce alert fatigue or suppress known issues directly from the chat.
List active silences
See what’s currently silenced to ensure you aren’t missing critical signals.
List all active silences Are there any silences for “cluster=prod-us-east-1”?
Create a silence
Silence an alert rule or a specific label set.
Silence the “High CPU Usage” alert for 2 hours Create a silence for “region=us-west-1” for 30 minutes Silence this alert until tomorrow morning
Debug routing
Troubleshoot notification delivery by inspecting your routing configuration.
Inspect notification policies
View the policy tree to understand how alerts are routed to contact points.
Show me the notification policy tree Which policy handles alerts with label “team=backend”?
Check contact points
List and inspect the configurations of your contact points to ensure integrations are set up correctly.
List all contact points Show details for the “Slack-Ops” contact point
Enrich alert rules
Add context to your alert rules to help responders investigate faster. You can attach log queries or specific Assistant instructions.
Add an enrichment
Attach a log query to an alert rule. When the alert fires, the Assistant can automatically run this query to show relevant logs.
Add a logs enrichment to the “High CPU” alert Create an enrichment for “Kubelet Down” with query “{app=‘kubelet’} |= ’error’”
List enrichments
See what enrichments are configured for a specific rule.
dependencies enriched for “High CPU Usage” List enrichments for the “Disk Space Low” rule
Check policies and contact points
Understand the policy tree and defined contact points.
Show me the notification policy tree List all contact points Who gets notified for alerts with “severity=critical”?
Add context to alerts
Make alerts more actionable by adding “enrichments.” Enrichments allow the Assistant or specific LogQL queries to automatically provide extra context when you ask about the alert later.
Add a log query enrichment to the “High CPU” rule Add an Assistant enrichment to “Kubelet down” with instructions to check node status
List enrichments
Check what enrichments are already configured for a rule:
Show me all the enrichments configured for my alert rule grafana-rule-abc123 What enrichments does the “High CPU” rule have?
Next steps
- Run an investigation to troubleshoot the source of an alert.
- Query data to explore metrics or logs related to a firing alert.
