Slide 3 of 4

The investigation workflow

Metrics investigation workflow

This pattern guides you from a broad view to specific findings. A typical investigation moves through five steps, though you can jump between them or skip the ones you don’t need.

Investigation workflow showing five steps

StepWhat you doWhat you get
SearchType part of a metric name in the search barList of matching metrics
FilterUse the filters to narrow scope by label valuesReduced set of time series
VisualizeView the automatic graph of the selected metricTime series visualization
InvestigateUse the Related metrics tab (metrics with similar names) and the Related logs tab (matching log streams)Related metrics, and a pivot from a metric to its logs
Act on itAdd to a dashboard, create an alert, save the view, copy the URL to share, or open in ExploreEach action carries your query and time range forward

You don’t need to know exactly what you’re looking for when you start.

You also don’t need to aggregate or apply rate functions. Filter to what you care about and read the graph, which reveals trends and anomalies that raw numbers hide. You can compare multiple time series side by side, for example CPU across servers, to spot correlations and narrow down a root cause.

Script

Let’s walk through a typical investigation in Grafana Metrics Drilldown. It’s a common pattern rather than a fixed sequence, so you can move between these steps as you go.

First, search for a metric. Type part of the name and Metrics Drilldown shows matching options. This helps when you’re not sure of the exact metric name.

Second, filter by labels. Select the label values that narrow down to what you care about. Instance, namespace, job, whatever dimensions matter for your investigation.

Third, visualize the data. Metrics Drilldown shows you the time series as a graph automatically, so you can see patterns, spikes, and trends without writing a query, aggregating, or applying rate functions. You can also compare multiple time series side by side, like CPU across servers, to spot correlations.

Fourth, investigate related data. The Related metrics tab surfaces other metrics with similar names and common prefixes, so you can find related metrics without knowing their exact names. The Related logs tab lets you pivot from a metric to the log streams that match it.

Finally, act on a useful finding. Add the panel to a dashboard or create an alert. Both arrive with your query and time range already filled in. You can also save the view to resume it later, copy its URL to share, or open it in Explore to write a custom query.

/
Previous Next