Grafana Cloud

Grafana Assistant Investigations

Use Grafana Assistant Investigations for incidents that span services or need analysis across metrics, logs, traces, and profiles. In Assistant Workspace, you provide the problem and review the Assistant’s hypotheses, evidence, and report.

For quick questions without a structured investigation, refer to Query data. Grafana Sift Investigation is a separate feature for automatic Kubernetes triage without a prompt.

Before you begin

Confirm that you have the following:

  • Grafana Assistant Investigations enabled in your Grafana Cloud stack
  • The Assistant Investigation User role and access to the data sources you need
  • A problem statement with the symptom, impact, affected services, and time range

Manual investigations use your identity and data source permissions. Automated investigations from alerting or IRM webhooks use service credentials and that service identity’s permissions. For access requirements, refer to Manage access (RBAC).

Start an investigation

Start in Workspace and describe what you want to investigate. The Assistant gathers evidence, tests hypotheses, and writes a report while you follow progress in the conversation.

  1. Open Assistant Workspace.
  2. Start a conversation and switch the mode to Investigation.
  3. Choose a private scope or add Grafana teams that need access.
  4. Leave the tier on Auto, or turn off Auto to choose Small, Medium, or Large.
  5. Send your problem statement. Add an image, for example, a chart screenshot, when it helps explain the issue.

Include specific services and a time range, for example:

Payment latency increased between 10:00 and 10:30 UTC today. Investigate @payment-cluster and check for database locks.

You can also ask the Assistant to check public provider status pages for outages that might explain your symptoms, for example:

Did AWS or Google Cloud report any outages in the last 24 hours that could explain our checkout latency?

You can also switch to Investigation in Assistant chat before sending your first prompt, then continue in Workspace. You can change the tier for later turns; a higher tier doesn’t guarantee a correct answer or a faster investigation. Investigations respect your stack’s monthly usage limits. For details, refer to Understand Grafana Assistant pricing and usage.

Review findings

Use the Workspace views to review the Assistant’s conclusions and their supporting evidence:

  • Report: Findings and recommended next steps
  • Hypotheses: Suspected causes, their status, and evidence
  • Sources: Generated panels and query output, when available

Click a source citation, for example, Source 1, to open its panel in Sources and inspect the query, time range, and data source. Hover over a citation for a preview. Click Expand diagram to inspect a report diagram in a larger view.

Before a report exists, select Disprove or Reopen on a hypothesis to draft a correction. Edit the message and send it to update the investigation’s direction. Expand Evidence or Timeline to review its support and history. Once a report exists, verdict controls replace Disprove and Reopen on hypotheses that make a causal claim. Answer Was this the cause? with Yes or No, and use Change to revise your verdict.

Send new evidence or a change in scope to continue a completed investigation. The Assistant updates the hypotheses and report when the findings change. Questions that only clarify or reformat existing findings leave the report unchanged.

Select Export report to copy the report, or open Download and choose Markdown (.md), HTML (.html), or PDF (.pdf). HTML creates a self-contained report with text, panels, and diagrams; PDF is available in a browser. Use the feedback controls to rate the investigation and explain what was useful or missing.

Interact with a running investigation

Ask questions or redirect the investigation while it works. You don’t need to start another investigation.

Ask a question without interrupting

Type /ask followed by a question, for example:

/ask give me an update on the current status

The answer opens above the conversation without changing the plan or report. It isn’t part of the investigation transcript. Workspace keeps recent answers so you can return to them.

In Assistant Workspace, you can also select text in Report or Hypotheses and click Ask to ask about the selection. Keep the conversation pane open to use this action. It isn’t available in the mobile layout. It uses the same answer view without changing the investigation’s direction.

Queue or steer follow-ups

Choose whether your message waits or redirects the investigation now:

  • Press Enter to queue a message. You can queue up to five messages, send one immediately, or remove it. With an empty input, Enter sends the first queued message
  • Press Cmd+Enter on macOS, or Ctrl+Enter on Windows and Linux, to stop the current work and steer the investigation right away. Other queued messages stay queued

Stop and review partial findings

Click Stop in Workspace or the Assistant sidebar. The Assistant stops gathering evidence and prepares a summary that marks the findings as incomplete. Keep the tab open until it finishes. If summary generation fails, a stopped notice appears instead.

Existing evidence and the report remain available. Send another message to continue with more context or a different direction. Stopping an investigation doesn’t stop independent repository coding agents.

Share an investigation

Use team scope for collaboration on the live investigation, or publish a snapshot for others in your Grafana organization.

Team members with the required investigation permissions can follow progress, ask questions, steer, and correct hypotheses in the same investigation. Team scope doesn’t replace those permissions. As the creator, use the scope picker to add or remove teams or return to a private scope. Only the creator can archive the investigation. The Assistant Admin role can view any investigation and its chat, including private ones.

For system-created investigations, you need team access or the Assistant System Investigation Viewer role. If Access denied names grafana-assistant-app.investigations.system:read, refer to Allow users to view system-created investigations.

Open Share conversation and choose how to share:

  • Copy link: Click Copy investigation link for the live investigation. Recipients must already have access; copying doesn’t grant access
  • Share with teams: Select teams and click Share with teams to add access. Use the scope picker to remove teams
  • Share a snapshot: Click Generate snapshot link to publish the transcript, report, hypotheses, source evidence, and your bookmarks. Anyone in your Grafana organization with the link can view it, regardless of team scope

Click Update to refresh an out-of-date snapshot or Stop sharing to revoke its link.

Slack investigation notifications also publish or refresh the snapshot automatically. Anyone in your Grafana organization with the snapshot link can view it, regardless of team scope. After you stop sharing, a later notification can publish a new snapshot link.

Tailor investigations to your environment

Give the Assistant context about your services and troubleshooting practices:

  • Infrastructure memory: Helps identify services, dependencies, and telemetry. Refer to Infrastructure memory
  • Rules: Instructions that always apply, for example, which metrics to prefer. Set Applications to Investigations. Refer to Configure Assistant rules
  • Skills: Runbooks and service-specific guidance the Assistant looks up when relevant. Turn on Visible to agents. Refer to Skills
  • MCP servers: Tools for other systems, for example, repositories that provide deployment context. Refer to MCP servers

To review the rules and skills an investigation uses, expand the investigation overview and click Applied context. Connecting a data source doesn’t make it part of every run; use rules or skills to explain when to query it.

Automate investigations

Start investigations automatically from incidents or alerts, or repeat checks in an existing investigation:

  • IRM webhooks: Create incident or alert-group webhooks and assign Grafana teams in Assistant > Settings > Investigations > Automatic: IRM. Findings return to the incident or alert group. Refer to Set up IRM webhooks
  • Alert rules: Add an investigation enrichment when editing a rule, or ask the Assistant to add it. Refer to Enrich alert rules and Continue investigations from alerts

Keep checking with automations

Tell the Assistant what to check, how often, and for how long, for example, “Check the storefront error rate every 10 minutes for two hours.”

You can also manage checks in the Workspace conversation:

text
/schedule create 10m check whether the storefront error rate has recovered
/schedule status
/schedule stop <AUTOMATION_ID>

Checks can run as often as once a minute, with up to five active automations per investigation. By default, they stop after 24 hours or 100 checks, whichever comes first. You can ask for a different duration or stop them earlier.

Use Automations in the investigation overview or the automation card in chat to review checks. Click View details to read the instruction or Stop to end a check. You need write access to create or stop checks. No organization-wide setting disables investigation automations.

Configure investigation settings

Open Assistant > Settings > Investigations to manage automated starts, continuation, and tools. You need grafana-assistant-app.settings.investigations:read to view settings and grafana-assistant-app.settings.investigations:write to change them. The Assistant Admin role includes both. IRM webhook management requires separate IRM permissions; refer to Configure IRM webhooks.

The page opens on Manual for Slack setup and investigation visibility. Select Automatic: IRM for webhook setup, IRM continuation, and IRM limits, or Automatic: Alerting for alert continuation and alert limits. Investigation tools appears below the tabs.

The main controls are:

  • Continue on alert updates and Continue on IRM updates: Resume completed investigations when significant alert changes or IRM updates arrive
  • Continuation window: How long alert changes continue an investigation instead of starting another one. Accepts 1 to 24 hours from creation, and defaults to 6 hours
  • Alert investigation limits and IRM alert investigation limits: Separate limits for new alert-enrichment and IRM alert-group investigations. Each defaults to 50 starts per 20 minutes. Severity-specific rules override All; remove all rules to allow unlimited starts. IRM incident starts and alert-group updates don’t count toward the IRM alert limit
  • IRM webhooks and Skipped incident labels: Manage webhooks, their teams, and label rules that exclude incidents
  • Query additional data sources: Allow other supported data sources alongside metrics, logs, traces, and profiles. On by default; data source permissions still apply. Refer to Query data for supported sources

Manage monthly investigation usage limits separately on Assistant > Usage.

Use investigations where you work

After you connect Slack and link your account, ask the Assistant to investigate from a channel. For setup, refer to Slack integration.

In Assistant > Settings > Investigations > Manual, set Investigation visibility to All my teams, Specific teams, or Only me, then click Save. This choice also applies to investigations you start from Microsoft Teams or the CLI. Until you save a choice, Slack investigations use All my teams, while Microsoft Teams and CLI investigations stay private. Teams explicitly assigned when an investigation starts take precedence over this preference. Workspace, alerting, and IRM investigations keep their own visibility rules.

Next steps