Slide 2 of 3

Queries

Queries

Queries tell the alert where to look for data, what metric to choose, and what time series to look at specifically.

Grafana Alerting can query many data sources, not only metrics. You can also alert on logs and other signals when the data source supports alerting. This journey focuses on metrics.

Two ways to write a query

The Prometheus query editor has two options:

  • Builder method: Lets you construct the query visually by picking the metric and operations from menus
  • Code method: For writing PromQL by hand, which is more powerful but harder if you’re new to the language

This journey starts with Builder because it’s the simplest way to build a query without learning PromQL. Everything Builder does maps to PromQL behind the scenes, so when you flip to the Code tab, you can see the generated PromQL query.

Overview of query components

You want to alert when the QuickPizza app returns server errors (HTTP 5xx responses). Your query will use these pieces:

  • The tutorial Prometheus data source:

    • Data source: grafanacloud-play-prom

  • A counter that tracks HTTP requests. Each time QuickPizza handles a request, this number grows, and each series carries a response status label:

    • Metric: quickpizza_server_http_requests_total

  • A label filter that matches every 5xx status code in one expression:

    • Label filter: status =~ "5.."

  • A way to count new errors over the last five minutes (counters only go up, so you need a range function), and optionally a sum so the alert uses one total across services:

    • Operation: Increase (over 5m), then optionally Sum

With the Builder method, that becomes a query equivalent to sum(increase(quickpizza_server_http_requests_total{status=~"5.."}[5m])).

The exact metric and labels depend on your data source. In the hands-on path, you build this query for the QuickPizza app. The steps you follow, including picking a data source, choosing a metric, filtering, transforming, and previewing, are the same steps you’ll use on your own Prometheus-compatible metrics later.

Always preview

Always preview your query before moving on. The preview is your proof the alert is watching real data.

For QuickPizza, the preview shows how many 5xx responses arrived in the last five minutes. If you skip previewing and the query is empty or wrong, the rule still saves, but it never fires on the condition you care about. You could walk away thinking the service is covered, and then during a real incident the alert you built stays silent.

Script

To build an alert, you first tell Grafana what data to watch. You will craft a query that tells Grafana what to pull and from where.

Grafana Alerting can query many data sources, not only metrics. You can also alert on logs and other signals from data sources that support alerting. This journey focuses on metrics with Prometheus.

A query has four pieces. You pick a data source, choose a metric, add label filters to narrow down to the time series you care about, and apply operations to transform the result.

The Prometheus query editor gives you two tabs for this. Builder lets you point and click, while Code is for writing PromQL manually. This journey starts with Builder because it’s the simplest way in.

In this journey, we use the QuickPizza demo application. Picture the QuickPizza app handling a stream of HTTP requests. You’d reach for a request counter, filter it to 5xx status codes — server errors — apply an increase over five minutes to count new errors in that window, optionally sum across services, and now you have something worth alerting on.

Always preview your query before moving on. If the preview returns empty, the alert has nothing to work with.

Previous Next