Documentation Grafana Cloud Knowledge graph Use cases Investigate incidents

Investigate incidents across multiple entities

Use RCA workbench to investigate incidents that span multiple services or infrastructure components. Correlate insights across entities on a timeline to understand what failed, when, and in what order.

When to use this workflow

Use this workflow when:

• Multiple services are experiencing errors at the same time
• You need to understand the sequence of events during an incident
• The root cause isn’t immediately obvious from a single service
• You want to correlate infrastructure issues with service degradation
• An alert fired but you need to understand the full scope of impact

This is your primary tool for incident response and root cause analysis.

Before you begin

Identify the services or infrastructure involved in the incident:

• Check the entity catalog for entities with critical insights
• Note the approximate time the incident started
• Have an initial hypothesis about which entities might be related

Open RCA workbench

From Grafana Cloud, navigate to Observability > RCA workbench.

Add entities to investigate

Gather all services and infrastructure involved in the incident into RCA workbench.

Add entities from the entity catalog

1. Navigate to Observability > Entity catalog.
2. Filter to entities with insights during the incident window.
3. Click each relevant entity.
4. Click Add to RCA workbench in the entity details.

Remove irrelevant entities

If you’ve added too many entities:

1. Switch to Timeline view.
2. Hover over an entity and click Delete entity from board (X icon).

Focus on entities directly involved in the incident.

Set the time range

Adjust the time range to focus on the incident window:

1. Use the time picker to select the incident period.
2. Start slightly before the first symptoms appeared.
3. Extend past when the incident resolved (or current time if ongoing).

A narrower time range makes patterns easier to spot.

Analyze the timeline

The Timeline view shows all insights chronologically across your selected entities.

Identify the first failure

1. Expand all entities on the left to show individual insights.
2. Scan the timeline from left to right.
3. Find the earliest insight that fired.

The first failure is often the root cause or trigger:

• Amend insight (blue) - Deployment or configuration change
• Failure insight (red) - Service or infrastructure failure
• Saturation insight (yellow/red) - Resource limit approached

Look for cascading failures

After identifying the first failure, trace its impact:

1. Note the time of the first insight.
2. Look for insights on other entities shortly after.
3. Identify the propagation pattern:
   • Downstream services start showing errors
   • Infrastructure saturation leads to Pod restarts
   • Database slowness causes service latency spikes

Use zoom to focus

Click and drag on the timeline to zoom into a specific time window:

1. Drag across the period of interest.
2. The view zooms to show more detail.
3. You may need to zoom multiple times for precise analysis.

To zoom out, use the time picker or click the reset zoom button.

Correlate insights with metrics

1. Click an insight on the timeline.
2. View the associated metric for that insight.
3. See how the metric crossed thresholds over time.
4. Compare with other entity metrics in the same window.

Example: Click an error rate breach insight to see the error rate spike visualized.

Investigate individual entities

From RCA workbench, drill into entities for more detail:

View entity details

1. Hover over an entity name in the timeline.
2. Click KPI.
3. The entity details drawer opens.
4. Switch between tabs:
   • Service overview - RED metrics and thresholds
   • Logs - Pre-filtered to the incident time range
   • Traces - Request traces during the incident
   • Kubernetes - Infrastructure health

Check logs for errors

1. Open the Logs tab for an entity.
2. Filter by severity: Error or Warning.
3. Look for stack traces or error messages at incident start time.
4. Search for specific error patterns.

Analyze slow traces

1. Open the Traces tab.
2. Look at the duration heatmap.
3. Identify traces that are slower than usual.
4. Click a slow trace to see:
   • Which service calls took longest
   • Database query times
   • External API call latency

Use the entity graph

Visualize relationships between affected entities:

1. From RCA workbench, click Graph view.
2. See all entities and their connections.
3. Identify:
   • Which service calls which (arrows show direction)
   • Upstream callers vs downstream dependencies
   • Infrastructure hosting the services

Navigate the graph

1. Click an entity in the graph.
2. View connected entities.
3. Add problematic connections to the timeline.
4. See if upstream or downstream services also have issues.

Common incident patterns

Recognize these common failure patterns to accelerate root cause identification.

Deployment triggered errors

Pattern: Amend insight (blue) immediately before error insights (red)

1. Find the Amend insight (deployment, scale event).
2. Note which service was deployed.
3. Check if error insights on that service or downstream started immediately after.
4. Review logs from the deployed service for startup errors.

Action: Likely a bad deployment. Rollback or investigate new code.

Resource saturation cascade

Pattern: Saturation insight followed by performance degradation across multiple services

1. Identify saturation insight (CPU, memory, connections).
2. See latency increases on the saturated service.
3. Upstream services show error rate increases (timeouts).
4. More services affected as the incident progresses.

Action: Scale resources, increase limits, or optimize the saturated service.

Database or dependency failure

Pattern: Multiple services show errors simultaneously, all calling the same dependency

1. Add multiple affected services to RCA workbench.
2. View the entity graph.
3. Identify shared downstream dependency.
4. Check if that dependency has failure insights.

Action: Investigate and restore the shared dependency.

Infrastructure failure impact

Pattern: Node or Pod failure followed by service errors

1. Add nodes and Pods to RCA workbench.
2. Find Pod restart or node NotReady insights.
3. Correlate with service error rate spikes.
4. Check if Pods running the service restarted.

Action: Fix infrastructure issue, ensure Pod rescheduling succeeds.

Use Grafana Assistant

Accelerate your analysis with Grafana Assistant:

1. In RCA workbench, click Analyze RCA workbench.
2. Grafana Assistant analyzes entities and timeline.
3. Ask questions like:
   • “What was the first failure?”
   • “Which entities are most affected?”
   • “What changed before the incident?”
   • “@payment-service what caused the error spike?”

Grafana Assistant can identify patterns and suggest root causes based on the timeline.

Document findings

Capture key information during your investigation to support post-incident analysis.

Key information to note

• Time of first failure - When did the incident actually start?
• Root cause entity - Which service or infrastructure component failed first?
• Triggering event - Deployment, configuration change, external dependency?
• Scope of impact - How many services/customers affected?
• Propagation pattern - How did the failure spread?

Create a summary

Document:

1. Root cause (what failed)
2. Trigger (why it failed)
3. Impact (scope and severity)
4. Timeline (sequence of events)
5. Resolution steps (what fixed it)

Next steps during incidents

Take appropriate action based on whether the incident is ongoing or resolved.

Incident is ongoing

• Mitigate: Rollback deployment, scale resources, or fail over
• Communicate: Share impact scope with stakeholders
• Monitor: Keep RCA workbench open to watch for new failures

Incident is resolved

• Post-mortem: Document root cause and timeline
• Prevent recurrence: Add monitoring, improve limits, or fix underlying issues
• Share learnings: Update runbooks and team knowledge

Related workflows

• Monitor services - Proactively catch issues before they become incidents
• Identify unhealthy infrastructure - Find infrastructure root causes
• Track changes - Identify deployments or configuration changes that triggered issues
• Explore dependencies - Understand service relationships to predict impact

Additional resources

• RCA workbench reference - Detailed feature documentation
• Grafana Assistant - AI-powered root cause analysis
• Entity graph - Visualize dependencies