Note
Grafana Assistant Investigations is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.
Run investigations
Grafana Assistant helps you investigate incidents by answering quick questions about your telemetry or running longer investigations in Assistant Workspace. Investigations explore metrics, logs, traces, and profiles, build hypotheses, and produce a report you can use during incident response.
Review investigations solutions in Grafana
An investigation can refer to multiple features in Grafana Cloud. This page covers Grafana Assistant Investigations. Here is how it differs from other investigation features:
- Grafana Assistant Investigations: Prompt-driven analysis that queries metrics, logs, traces, and profiles across your Grafana Cloud data. It lives in Assistant Workspace and produces a structured report with hypotheses and source queries.
- Grafana Sift Investigation: ML-powered, automatic analysis of Kubernetes infrastructure. Runs curated detectors over cluster signals without a prompt. Free in Grafana Cloud. Not part of Grafana Assistant.
Choose the right tool for the job:
- Use Assistant Investigations for cross-signal, service-level analysis guided by your prompt.
- Use Sift Investigation for Kubernetes-only issues where you want quick, automatic triage.
Before you begin
- Investigations entitlement: Enable Grafana Assistant Investigations in Grafana Cloud.
- Investigation access: Assign the Assistant Investigation User role if your environment uses RBAC.
- Incident context: Summarize the symptom, impact, and affected services before launching an investigation.
Review permissions
Grafana Assistant runs investigations on your behalf. The Assistant can only access the data sources and resources that you have permission to view.
Access controls include:
- User identity: The Assistant uses your identity to execute queries.
- Data access: Investigations can only query metrics, logs, traces, and profiles that you are authorized to access.
- RBAC compliance: All investigation activities respect your organization’s existing role-based access control policies.
- Investigation visibility: You can see investigations you created, investigations that include one of your Grafana teams in their scope, and deprecated investigation rows. System-created investigations from IRM webhooks, alerts, or incidents require either team access or the Assistant System Investigation Viewer role. For more information, refer to Manage access (RBAC).
Build infrastructure context
To provide accurate answers, the Assistant needs to understand your specific environment. Infrastructure memory automatically scans your data sources and builds knowledge about your services, namespaces, and dependencies. Once available, the Assistant can map natural language queries like “How is the checkout service?” to the correct metrics and labels in your system. For more information, refer to Infrastructure memory.
Leverage dashboard context
Grafana Assistant automatically uses your dashboards to understand how your services are monitored. When you ask a question or launch an investigation, the Assistant scans your recent dashboards to find relevant panels, queries, and variables.
This helps the Assistant:
- Identify key metrics: Discover the specific metric names and labels used in your dashboards.
- Understand topology: Learn how services relate to each other based on dashboard links.
- Find logs and traces: Use the queries in your logs and traces panels as a starting point.
Ask quick questions
Use the chat interface to get immediate answers about system health without writing queries manually. Context is key, mention specific resources to get the best results.
Ask about specific signals
Query metrics directly by mentioning the service or data source.
Show the error rate for @checkout-service over the last hour.
Filter logs
Search for specific patterns in your logs within a time range.
Find logs mentioning ’timeout’ in @loki-prod from 10:00 to 10:15.
Refine the answer
Iterate on the results to group or sort the data.
Group the results by pod_name.
Correlate multiple signals
The Assistant can help you verify hypotheses by checking different types of data across the same timeframe.
Establish a baseline
Start with a metric or dashboard panel to set the context.
Look at the CPU usage on this panel.
Pivot to other data
Ask the Assistant to find related logs or traces for the same timeframe.
Are there any error logs for the same service during that CPU spike?
Synthesize
Ask for a summary that connects the findings across signals.
Explain how the CPU spike relates to the error logs.
Start an investigation
For complex incidents, use Investigation mode. This starts a longer-running Assistant investigation in Workspace. The investigation can analyze multiple data sources, test hypotheses, and generate a structured report while you continue to review progress in the conversation.
When to launch an investigation
Use Investigation when issues span multiple services, require more than one signal type, or when you need a structured report for an incident.
You can also automate investigations using IRM webhooks. When configured, the Assistant automatically starts investigations when alert groups are created or incidents are updated.
Note
If you use automated investigations from alerting or IRM, configure the Grafana team for the webhook in Assistant > Settings > Integrations > IRM webhooks. Team assignment controls who can see investigations that those webhooks create.
Understand the investigation lifecycle
- Launch: Provide a detailed prompt that captures the incident summary, timeframe, affected services, and focus areas.
- Plan: The Assistant creates and updates hypotheses as it learns more about the problem.
- Gather evidence: The Assistant queries available metrics, logs, traces, and profiles to confirm or rule out hypotheses.
- Update progress: The Workspace conversation shows progress and lets you add hints, corrections, or follow-up questions.
- Report: The Assistant produces a report with key findings, supporting evidence, and recommended next steps.
Investigations track token usage separately from chat and respect the monthly tenant limits defined in Grafana Cloud.
Run an assistant investigation
Open Assistant Workspace.
Start a conversation and switch the mode to Investigation.
If you want teammates to see the investigation, choose the investigation scope before you send the prompt.
Provide a detailed problem statement.
High latency in the payment service. Investigate the @payment-cluster and check for database locks.
Monitor progress in Workspace.
Review the report, hypotheses, and source queries.
You can also start from Assistant chat by switching the mode to Investigation before you send the first prompt. The Assistant creates an investigation conversation that you can continue in Workspace.
Working with investigation reports
The Report view contains the investigation findings and recommended next steps. As the investigation progresses, the Assistant updates the report and can add diagrams and tables.
When the investigation has a hypothesis plan, Workspace shows the Hypotheses view. Use it to understand what the Assistant is checking, what it has ruled out, and what remains open.
When the investigation generates panel-based query output, Workspace shows the Sources view. Use it to inspect generated panels and query output that support the report.
Open the investigation in a full-page view when you need more space to read the report or inspect hypotheses. When Workspace shows Regenerate report, use it to rebuild the report from the current investigation state.
When you are ready to communicate, ask the Assistant to convert the findings into incident updates, backlog items, or dashboard follow-ups.
Review deprecated investigations
The investigations list can include Deprecated rows. These rows use the investigation workbook view.
You can open deprecated investigations to review their content. To start an investigation, use Investigation mode in Workspace or Assistant chat.
