Manage and investigate alerts
Grafana Assistant interacts with Grafana Alerting to help you find firing alerts, understand their history, create and update Grafana-managed alert rules, create and change ratio SLOs, and manage silences without leaving the chat. You can also inspect routing policies and add context to alert rules to speed up future investigations.
What you’ll achieve
- Find and inspect alert rules instantly by name or status.
- Create, update, and delete Grafana-managed alert rules after review and approval.
- Create, update, and delete ratio SLOs after review and approval.
- Review alert state history to see when alerts fired or resolved.
- Manage silences to reduce noise during maintenance.
- Audit routing configuration to ensure the right teams are notified.
- Enrich alert rules with automated Assistant context.
Before you begin
- Grafana Alerting must be enabled.
- Grafana-managed alert rules only: Assistant lists and inspects Grafana-managed alert rules. To review data source-managed rules, for example, rules stored in Mimir or Loki, open Alerting > Alert rules in Grafana.
- Provisioned rules stay in their source of truth: If a rule is managed in Terraform, OpenTofu, Pulumi, or another provisioning source, Assistant won’t change it. Update the owning source instead.
- Ratio SLOs only: Assistant creates and updates ratio SLOs only.
- Grafana SLO access: To create or edit a ratio SLO, Grafana SLO must be installed, the destination data source must be a Grafana Prometheus data source, and you must be able to edit the target folder.
- To list and inspect rules and policies, your user account requires Viewer permissions or higher.
- To create silences or modify rules, you must have Editor or Admin permissions.
Create and change alert rules
Use Assistant when you want to draft or change a Grafana-managed alert rule in chat, then review and approve the change before Assistant applies it.
Create an alert rule
Ask Assistant for the alert you want, then review the proposed query, condition, duration, labels, and routing.
Create a warning alert rule for checkout latency above 500ms for 10 minutes Create a critical alert when payment error rate is above 5% for 5 minutes and route it to the platform team
Assistant asks you to approve the change before it creates the rule.
Update an alert rule
Ask Assistant to change an existing rule when you want to adjust thresholds, labels, timing, or routing.
Increase the threshold on the “High CPU Usage” alert rule to 90% Update the “Checkout latency” rule to wait 10 minutes before it fires Change the severity label on the “Disk Space Low” alert to warning
Assistant keeps the rest of the rule unchanged unless you ask to replace it.
Delete an alert rule
Delete a Grafana-managed rule when it is obsolete or has been replaced.
Delete the “Old synthetic check” alert rule Remove alert rule UID grafana-rule-abc123
Assistant asks you to approve the deletion before it removes the rule.
Create and change ratio SLOs
Use Alerts & Watchers mode when you want Assistant to help you manage a ratio SLO and discuss whether you also need an alert or a Watcher.
Create a ratio SLO
Ask Assistant for the SLO you want, then review the proposed name, success metric, total metric, objective, window, and folder before you approve the change.
Create a 99.5% checkout availability SLO over 28 days using checkout success and total request counters Create a billing API latency SLO in the platform folder with a 99.9% objective
Assistant creates ratio SLOs only. It doesn’t create alerts from the SLO. After you create the SLO, configure its alerts in Grafana SLO.
Update or delete a ratio SLO
Ask Assistant to change selected SLO fields, or remove an SLO that you no longer need.
Update the checkout availability SLO to 99.9% Move the billing latency SLO to the team-services folder Delete the SLO with UUID 123e4567-e89b-12d3-a456-426614174000
Assistant asks you to approve every SLO write before it applies the change.
Know when Assistant won’t write the change
Assistant refuses to update or delete rules that show provisioning ownership metadata, because those rules are managed outside Grafana.
If ownership is unclear, Assistant can still help you inspect the rule and prepare the change you need to make in the owning repository.
Assistant also refuses to update or delete SLOs that are provisioned by Terraform, file provisioning, or another as-code source. Change those SLOs in the owning source so Grafana doesn’t drift from it.
Explore and inspect alerts
Quickly find what’s firing or look up specific rules to understand their configuration.
List and filter alerts
Ask the Assistant to show you alerts based on their state or name.
Show me all firing alerts Show me firing alerts in the “Production” folder Find alert rules related to “cpu usage”
Inspect rule details
Get a comprehensive view of a rule’s configuration, including its query, threshold, and current health.
Show details for the “High CPU Usage” alert rule What is the configuration of the “Kubelet down” alert?
View version history
Track changes to alert rules over time to correlate modifications with incidents.
Show me the version history for the “High CPU Usage” rule Who last modified the “Disk Space Low” alert?
Investigate alert history
Understand the behavior of your alerts over time. The Assistant can query the alert historian to show state transitions and notifications.
Analyze state transitions
See exactly when an alert moved from Normal to Pending or Firing.
When did “High CPU Usage” last fire? Show me the alert state history for “High CPU Usage” for the last 24 hours
Check notifications
Verify if and when notifications were sent for a specific alert.
Show me the notification history for the “High CPU Usage” alert Did the “Disk Full” alert send a notification yesterday?
Manage silences
Reduce alert fatigue or suppress known issues directly from the chat.
List active silences
See what’s currently silenced to ensure you aren’t missing critical signals.
List all active silences Are there any silences for “cluster=prod-us-east-1”?
Create a silence
Silence an alert rule or a specific label set.
Silence the “High CPU Usage” alert for 2 hours Create a silence for “region=us-west-1” for 30 minutes Silence this alert until tomorrow morning
Debug routing
Troubleshoot notification delivery by inspecting your routing configuration.
Inspect notification policies
View the policy tree to understand how alerts are routed to contact points.
Show me the notification policy tree Which policy handles alerts with label “team=backend”?
Check contact points
List and inspect the configurations of your contact points to ensure integrations are set up correctly.
List all contact points Show details for the “Slack-Ops” contact point
Enrich alert rules
Add context to your alert rules to help responders investigate faster. You can attach log queries or specific Assistant instructions.
Add an enrichment
Attach a log query to an alert rule. When the alert fires, the Assistant can automatically run this query to show relevant logs.
Add a logs enrichment to the “High CPU” alert Create an enrichment for “Kubelet Down” with query “{app=‘kubelet’} |= ’error’”
List enrichments
See what enrichments are configured for a specific rule.
dependencies enriched for “High CPU Usage” List enrichments for the “Disk Space Low” rule
Check policies and contact points
Understand the policy tree and defined contact points.
Show me the notification policy tree List all contact points Who gets notified for alerts with “severity=critical”?
Add context to alerts
Make alerts more actionable by adding “enrichments.” Enrichments allow the Assistant or specific LogQL queries to automatically provide extra context when you ask about the alert later.
Add a log query enrichment to the “High CPU” rule Add an Assistant enrichment to “Kubelet down” with instructions to check node status
List enrichments
Check what enrichments are already configured for a rule:
Show me all the enrichments configured for my alert rule grafana-rule-abc123 What enrichments does the “High CPU” rule have?
Next steps
- Assistant Investigations to troubleshoot the source of an alert.
- Query data to explore metrics or logs related to a firing alert.
Was this page helpful?
Related resources from Grafana Labs
