Documentation Grafana Cloud Observe and act Test and validate Synthetic Monitoring Analyze results Custom labels

Custom labels

Synthetic Monitoring allows you to define custom labels in your check definitions. Use custom labels to organize and filter your checks, query check results by team, environment, or service, and route alerts to the right receivers.

You can add up to 10 custom labels to a check in the Labels step of the check editor. You can also add up to 3 custom labels to your private probes.

How custom labels appear on your data

Custom labels appear as you write them on all metrics and logs that a check generates. For example, if you add a custom label environment with the value prod to a check, every metric from that check carries environment="prod":

promql Copy

probe_duration_seconds{job="example", instance="https://www.grafana.com/", environment="prod"}

You can’t use a label name that Synthetic Monitoring reserves for its own use, such as job, instance, or probe. For the full list, refer to System labels.

Query check results by custom labels

Because custom labels are part of each metric’s label set, you can filter any Synthetic Monitoring metric by them directly:

promql Copy

probe_success{environment="prod"}

The same applies to check logs. Custom labels are indexed on your log streams, so you can use them in LogQL stream selectors:

logql Copy

{source="synthetic-monitoring-agent", environment="prod"}

Alert on custom labels

Custom labels appear on alert instances for any alert rule that evaluates your check metrics, which means you can use them in alert routing without extra work.

For example, this alert rule expression fires if the duration of any probe for a check labeled environment="prod" exceeds 250 ms:

promql Copy

probe_duration_seconds{environment="prod"} > 0.250

The resulting alert instances carry environment="prod", so a notification policy that matches environment=prod routes them to the right team. For more information about alerting on check results, refer to Configure alerts.

Checks with the label_ prefix (legacy)

This section applies to tenants that haven’t migrated to un-prefixed labels yet. It’s also useful when you query historical data: metrics recorded before your migration keep the prefixed label names for their full retention period.

Before migration, custom labels behave differently in two ways:

• Custom labels are applied only to the sm_check_info metric and to check logs, not to the other metrics a check generates.
• Custom label names carry a label_ prefix. A custom label environment appears as label_environment.

To see the details of checks with a given custom label, query sm_check_info directly:

promql Copy

sm_check_info{label_environment="prod"}

To use custom labels with any other metric, join the metric with sm_check_info using a Prometheus join on the labels shared by all Synthetic Monitoring metrics: job, instance, probe, and config_version.

Example

This example propagates custom labels onto the probe_duration_seconds metric and alerts if the duration of a probe exceeds 250 ms:

promql Copy

sum without (config_version, frequency, geohash) (
  probe_duration_seconds{job="example", instance="https://www.grafana.com/"}
  *
  on (job, instance, probe, config_version)
  group_right ()
  sm_check_info{job="example", instance="https://www.grafana.com/"}
)
> 0.250

Unpacking the expression:

• sm_check_info always has the value 1, so multiplying by it leaves the metric values unchanged while the join adds the labels.
• on (job, instance, probe, config_version) joins the two metrics on the labels every Synthetic Monitoring metric shares.
• group_right () copies the extra labels from sm_check_info, including your label_-prefixed custom labels, onto the result.
• sum without (config_version, frequency, geohash) drops labels whose values change when you edit the check, so the resulting series stay continuous across configuration changes.

The result carries label_environment="prod", which you can use for alert routing or dashboard filtering.

To filter by a custom label, apply the filter to the sm_check_info side of the join:

promql Copy

sum without (config_version, frequency, geohash) (
  probe_duration_seconds{}
  *
  on (job, instance, probe, config_version)
  group_right ()
  sm_check_info{label_environment="prod"}
)
> 0.250

This produces a time series per probe for every check that has that label, without naming each check individually.

To stop needing these joins and use your labels as written, refer to Migrate to un-prefixed labels.