Grafana Assistant Investigations
Grafana Assistant helps you investigate incidents by answering quick questions about your telemetry or running longer investigations in Assistant Workspace. Investigations explore metrics, logs, traces, and profiles, build hypotheses, and produce a report you can use during incident response.
Review investigations solutions in Grafana
An investigation can refer to multiple features in Grafana Cloud. This page covers Grafana Assistant Investigations. Here is how it differs from other investigation features:
- Grafana Assistant Investigations: Prompt-driven analysis that queries metrics, logs, traces, and profiles across your Grafana Cloud data. It lives in Assistant Workspace and produces a structured report with hypotheses and source queries.
- Grafana Sift Investigation: ML-powered, automatic analysis of Kubernetes infrastructure. Runs curated detectors over cluster signals without a prompt. Free in Grafana Cloud. Not part of Grafana Assistant.
Choose the right tool for the job:
- Use Assistant Investigations for cross-signal, service-level analysis guided by your prompt.
- Use Sift Investigation for Kubernetes-only issues where you want quick, automatic triage.
Before you begin
- Investigations entitlement: Enable Grafana Assistant Investigations in Grafana Cloud.
- Investigation access: Assign the Assistant Investigation User role if your environment uses RBAC.
- Incident context: Summarize the symptom, impact, and affected services before launching an investigation.
Review permissions
Grafana Assistant runs investigations on your behalf. The Assistant can only access the data sources and resources that you have permission to view.
Access controls include:
- User identity: The Assistant uses your identity to execute queries.
- Data access: Investigations can only query metrics, logs, traces, and profiles that you are authorized to access.
- RBAC compliance: All investigation activities respect your organization’s existing role-based access control policies.
- Investigation visibility: You can see investigations you created, investigations that include one of your Grafana teams in their scope, and deprecated investigation rows. System-created investigations from IRM webhooks, alerts, or incidents require either team access or the Assistant System Investigation Viewer role. For more information, refer to Manage access (RBAC).
Build infrastructure context
To provide accurate answers, the Assistant needs to understand your specific environment. Infrastructure memory automatically scans your data sources and builds knowledge about your services, namespaces, and dependencies. Once available, the Assistant can map natural language queries like “How is the checkout service?” to the correct metrics and labels in your system. For more information, refer to Infrastructure memory.
Leverage dashboard context
Grafana Assistant automatically uses your dashboards to understand how your services are monitored. When you ask a question or launch an investigation, the Assistant scans your recent dashboards to find relevant panels, queries, and variables.
This helps the Assistant:
- Identify key metrics: Discover the specific metric names and labels used in your dashboards.
- Understand topology: Learn how services relate to each other based on dashboard links.
- Find logs and traces: Use the queries in your logs and traces panels as a starting point.
Ask quick questions
Use the chat interface to get immediate answers about system health without writing queries manually. Context is key, mention specific resources to get the best results.
Ask about specific signals
Query metrics directly by mentioning the service or data source.
Show the error rate for @checkout-service over the last hour.
Filter logs
Search for specific patterns in your logs within a time range.
Find logs mentioning ’timeout’ in @loki-prod from 10:00 to 10:15.
Refine the answer
Iterate on the results to group or sort the data.
Group the results by pod_name.
Correlate multiple signals
The Assistant can help you verify hypotheses by checking different types of data across the same timeframe.
Establish a baseline
Start with a metric or dashboard panel to set the context.
Look at the CPU usage on this panel.
Pivot to other data
Ask the Assistant to find related logs or traces for the same timeframe.
Are there any error logs for the same service during that CPU spike?
Synthesize
Ask for a summary that connects the findings across signals.
Explain how the CPU spike relates to the error logs.
Check cloud-provider status pages
The Assistant can check recent public status-page events for cloud providers during incident triage. This helps you determine whether an AWS, Azure, Google Cloud, Oracle Cloud, or Anthropic service event might explain a symptom.
Check cloud provider status pages for incidents in the last 24 hours.
Did AWS or Google Cloud report any outages that could explain our checkout latency?
Start an investigation
For complex incidents, use Investigation mode. This starts a longer-running Assistant investigation in Workspace. The investigation can analyze multiple data sources, test hypotheses, and generate a structured report while you continue to review progress in the conversation.
When to launch an investigation
Use Investigation when issues span multiple services, require more than one signal type, or when you need a structured report for an incident.
You can also automate investigations using IRM webhooks. When configured, the Assistant automatically starts investigations when alert groups are created or incidents are updated.
Note
If you use automated investigations from alerting or IRM, configure the Grafana team for the webhook in Assistant > Settings > Investigations. Team assignment controls who can see investigations that those webhooks create.
Understand the investigation lifecycle
- Launch: Provide a detailed prompt that captures the incident summary, timeframe, affected services, and focus areas.
- Plan: The Assistant creates and updates hypotheses as it learns more about the problem.
- Gather evidence: The Assistant queries available metrics, logs, traces, and profiles to confirm or rule out hypotheses.
- Update progress: The Workspace conversation shows progress and lets you add hints, corrections, or follow-up questions.
- Report: The Assistant produces a report with key findings, supporting evidence, and recommended next steps.
Investigations track token usage separately from chat and respect the monthly tenant limits defined in Grafana Cloud.
Run an assistant investigation
Open Assistant Workspace.
Start a conversation and switch the mode to Investigation.
If you want teammates to work on the investigation with you, choose the investigation scope before you send the prompt. Anyone on a team in the scope can open the investigation, ask questions, steer it, and challenge its hypotheses.
Provide a detailed problem statement. You can paste or attach an image, such as a chart screenshot or incident timeline, when visual context helps describe the issue.
High latency in the payment service. Investigate the @payment-cluster and check for database locks.
Monitor progress in Workspace.
Review the report, hypotheses, and source queries.
You can also start from Assistant chat by switching the mode to Investigation before you send the first prompt. The Assistant creates an investigation conversation that you can continue in Workspace.
Share an investigation and manage visibility
You control who can reach an investigation in two ways: the scope, which grants ongoing access, and a share link, which captures a snapshot.
The investigation scope controls who can find and open the live investigation. If you created it, use the scope picker in the investigation conversation to add or remove Grafana teams after it starts, or to return the investigation to a private scope. Keep the right responders included as ownership or incident participation changes.
To hand someone a fixed view of your findings, select Share conversation and then Generate share link. The snapshot includes the conversation transcript, the report, and the hypotheses as they were when you generated it. It’s only accessible within your Grafana organization, and later conversation isn’t added automatically. When the investigation moves on, the dialog reports the snapshot as out of date and offers Update to refresh it. Select Stop sharing to revoke the link.
The share dialog also has a Share investigation with additional teams section, which adds teams to the scope. Scope widening is one-way, so you can add teams here but not remove them.
Working with investigation reports
The Report view contains the investigation findings and recommended next steps. As the investigation progresses, the Assistant updates the report and can add diagrams and tables.
When the investigation has a hypothesis plan, Workspace shows the Hypotheses view. Use it to understand what the Assistant is checking, what it has ruled out, and what remains open.
When the investigation generates panel-based query output, Workspace shows the Sources view. Use it to inspect generated panels and query output that support the report.
The investigation overview also shows applied rules and skills when they influenced the investigation. Use these chips to understand which reusable guidance the Assistant followed while it planned and gathered evidence.
Follow inline source citations
As the investigation progresses, the Assistant annotates its findings with inline source citations. These appear as numbered chips — for example, Source 1 — directly in the chat messages and in the report. Each chip links to the panel query that produced the data the Assistant used to reach that conclusion.
To follow a citation, click the chip. Workspace opens the referenced panel in the Sources view where you can inspect the query, time range, and datasource. Hovering over a chip shows a preview of the panel.
Source citations only appear in investigation conversations. In standard Assistant chat, any citation-style markers in a response appear as plain text.
Open the investigation in a full-page view when you need more space to read the report or inspect hypotheses. From the report export menu, you can copy the report, download it as Markdown, or select Save as PDF to create a shareable report with text, panels, and diagrams.
After an investigation completes, provide feedback when prompted or use the feedback controls. Quick ratings help improve future investigations and the full feedback form lets you add details about what was useful or missing.
When you are ready to communicate, ask the Assistant to convert the findings into incident updates, backlog items, or dashboard follow-ups. Recommended next steps can include confidence indicators, and completed investigations can suggest follow-up prompts so you can continue the analysis without writing a new question from scratch.
Interact with a running investigation
You don’t have to wait for an investigation to finish. While it runs, you can ask it questions, correct its direction, and challenge its conclusions without starting over.
Ask a question without interrupting
Type /ask followed by a question to get an answer alongside the investigation while it keeps working.
/ask give me an update on the current status
The answer opens in a panel above the conversation. The Assistant can read your data to answer, but it can’t change the plan or the report, and the answer isn’t added to the investigation transcript. Workspace keeps your recent answers so you can page back through them. Use /ask for status checks and side questions when you don’t want to disturb the run.
Queue or steer follow-ups
When you send a message while the investigation is working, you choose whether it waits or takes effect now.
Press Enter to queue the message. The send button reads Queue, and queued messages run in order as the investigation reaches them. You can queue up to five messages, send one immediately, or remove it. With an empty input, Enter sends the first queued message.
Press Cmd+Enter, or Ctrl+Enter on Windows and Linux, to steer instead. Steering stops what the Assistant is doing and redirects it right away, so use it when the current line of work is wrong. Anything already queued stays queued.
Disprove or reopen a hypothesis
When you know something the investigation doesn’t, correct its plan directly. On a hypothesis card, select Disprove, or Reopen on a hypothesis it already ruled out.
Either action drafts a message in the composer. Edit it to add what you know, then send it. The Assistant updates the hypothesis and re-plans around what’s left, so a single correction can redirect the rest of the investigation.
Each hypothesis carries a status of open, root cause, symptom, disproven, or blocked. Expand Hypothesis timeline on a card to see when the hypothesis was added and what evidence changed it. Select a row to jump to the step that produced it.
Investigate as a team
When a Grafana team is in the investigation scope, everyone on that team works in the same run rather than starting their own. Teammates see progress live, and their follow-ups, steers, and hypothesis corrections all apply to the shared investigation. Only the creator can change the scope or archive the investigation.
Tailor investigations to your environment
Investigations use the same customization as the rest of Grafana Assistant. Which one you want depends on whether your guidance always applies, applies in specific situations, or adds new tools:
- Rules: Standing preferences and instructions, such as which metric to prefer or how to report findings. To make a rule apply to investigations, set Applications to Investigations. For more information, refer to Configure Assistant rules.
- Skills: Knowledge the Assistant looks up when it’s relevant, which makes skills a good home for runbooks and service-specific troubleshooting steps. An investigation can only discover a skill when Visible to agents is on. For more information, refer to Skills.
- MCP servers: Extra tools an investigation can use, which turns other systems into evidence. Connecting the GitHub MCP server, for example, lets an investigation read your repositories and relate a code or deployment change to what it sees in your telemetry. For more information, refer to MCP servers.
Automate investigations
You can have investigations start without anyone opening Workspace, and have them keep checking after they report.
Trigger investigations from IRM
Create webhooks in Assistant > Settings > Investigations. Incident webhooks start an investigation when an incident is declared, and alert group webhooks start one when an alert group fires in an integration you select. When the investigation finishes, it posts its summary back to the incident or alert group, with a link to the full investigation.
Assign Grafana teams to a webhook to control who can see the investigations it creates. Without teams, the Assistant falls back to teams from the alert or incident metadata. You can also skip incidents by label so routine work doesn’t trigger an investigation.
Investigations only start when there’s enough context on the incident, so some incidents won’t trigger one. For more information, refer to Set up IRM webhooks.
Trigger investigations from alert rules
You can attach an investigation to an alert rule as an enrichment, so the investigation runs when the rule fires and its findings travel with the alert. Ask the Assistant to add the enrichment for you, or add it while editing the rule. For more information, refer to Enrich alert rules.
Repeat and changed notifications for the same alert group continue the existing investigation instead of starting another one. For more information, refer to Continue investigations from alerts.
Keep checking with loops
A loop re-sends a prompt to an investigation on a schedule, which lets it watch a situation instead of concluding on state that’s still moving.
An investigation can open a loop itself when it needs to see whether something changes, and close it once the situation resolves. You can also create one with /loop create <interval> <what to re-check>, list them with /loop status, and end one with /loop stop <id>. Checks run about once a minute at the fastest, so the shortest useful interval is 60 seconds.
/loop create 10m check whether the storefront error rate has recovered
Loops stop when the investigation stops them, when you stop them, or when they reach a limit on checks or watch time. An investigation can have up to five active loops.
Configure investigation settings
Organization-wide investigation behavior lives in Assistant > Settings > Investigations, and needs the Assistant Admin role. Investigation usage limits are set separately on the Usage page.
The settings are:
- Enable loops: Allow investigations to run scheduled checks. When off, investigations can’t create loops and existing loops pause until you re-enable them.
- Continue investigations on alert updates: Add later alert-group changes to a completed investigation’s context, and continue the investigation when the changes are significant.
- Continuation window (hours): How long after an investigation is created that alert changes continue it instead of starting a new investigation. Match this to your notification policy’s repeat interval. Accepts 1 to 24 hours, and defaults to 6.
- Continue investigations on IRM updates: Resume a completed investigation when its incident or alert group changes.
- IRM webhooks: Create, disable, and delete the webhooks that start investigations, and assign teams to them.
- Skipped incident labels: Define key-value rules so incidents with matching labels don’t trigger investigations.
Use investigations where you work
Investigations don’t have to start or end in Workspace.
In Slack, ask the Assistant to look into a problem from the channel you’re already in. Choose who can see the investigations you start there under Who can see these investigations? in Slack setup. That setting only covers investigations you start from Slack. Investigations from alerting, IRM webhooks, or Workspace keep their own visibility rules. For more information, refer to Slack integration.
Investigations that IRM triggers post their findings back to the incident or alert group, so if you mirror IRM to Slack, the summary and its link reach the channel without any extra setup.
Review deprecated investigations
The investigations list can include Deprecated rows. These rows use the investigation workbook view.
You can open deprecated investigations to review their content. To start an investigation, use Investigation mode in Workspace or Assistant chat.
