Documentation Grafana Cloud Observe and act Test and validate Synthetic Monitoring Migrate to un-prefixed labels

Migrate to un-prefixed labels in Synthetic Monitoring

Synthetic Monitoring is changing how custom labels work to match the rest of Grafana Cloud:

• Custom labels appear on check metrics. Labels you define on a check are attached to all metrics the check generates, not only to the sm_check_info metric. This removes the need for PromQL joins, and enables cost attribution and label-based workflows on check metrics.
• The label_ prefix is removed. Labels appear as you write them. For example, a label named team appears as team, not label_team. Alert routing, dashboards, and queries behave the way you expect across Grafana Cloud.

This page explains the migration states, what to expect at each transition, and the steps to migrate safely.

If your checks don’t use custom labels, nothing changes and no action is needed. If they do, read on: anything that references your labels by their prefixed names needs to be updated during the migration.

Key dates

Migrate before the automatic migration deadline to control the timing yourself. For details, refer to What happens if you don’t migrate.

Migration states

The migration moves your tenant through three states. Dual-write is a safe middle state that lets you update your configuration at your own pace while both label forms are available.

Tenants created after the feature launch start in the un-prefixed state and don’t need to migrate.

What to expect when you migrate

When you enable dual-write

Enabling dual-write is the more disruptive of the two transitions, because it adds labels to your check metrics.

A series’ identity in Prometheus is its full label set. When labels are added to your check metrics, each affected series is replaced by a new one: the old series stops receiving samples and drops out of instant queries within about five minutes, and the new series begins with your next check execution.

In practice:

• Dashboards filtered by custom labels show a break at the transition. Historical data remains queryable under the old label names.
• Alerts firing on affected series resolve and re-fire as new alert instances, with for timers reset. You may see resolve and re-fire notification pairs.
• SLOs re-evaluate against the new series. Burn-rate windows that span the transition may show a brief blip.
• Aggregations over the job, instance, and probe labels remain continuous.

Treat the transition as a brief maintenance window: inform your on-call team and consider silencing Synthetic Monitoring alerts for 10 to 15 minutes around the state change.

When you finalize

Finalizing to the un-prefixed state is a much smaller event. Your check metrics keep the same label set, so alerts, SLOs, and dashboards built on them aren’t disturbed again. Only sm_check_info and check logs change: the prefixed label copies are removed.

Queries, recording rules, and alert routing policies that still reference label_-prefixed names stop matching new data at this point.

Reserved labels

As part of this migration, Synthetic Monitoring reserves a set of label names for system use, such as job, instance, and probe. Enforcement begins when you start migrating:

• Before you can enable dual-write, any custom label that collides with a reserved name must be renamed. The API and in-app warnings flag these collisions, and the migration can’t start until they’re resolved.
• After you’ve started migrating, the API rejects reserved label names when you create or update checks and probes.
• If a check reaches the un-prefixed state with an unresolved collision, Synthetic Monitoring emits the system value on the affected metrics and logs so the telemetry pipeline and app dashboards keep working. Your custom value stops appearing on those series until you rename the label.
• If Synthetic Monitoring reserves a new label in the future that you’re already using, your existing checks continue to run with your label.

For the list of reserved labels and enforcement details, refer to System labels.

Migration checklist

During dual-write, audit the following items. They commonly reference labels and need to be forward-compatible before you finalize. Work through them in this order:

1. Alert notification policies. For each matcher on a prefixed label, add or confirm the un-prefixed equivalent. For example, a routing policy that matches label_service=checkout needs a matcher for service=checkout. If you run parallel policy trees for both forms, keep both during dual-write and remove the duplicate tree after you finalize.

   Note

   While dual-writing, alerts may match both policy trees, so you may see duplicate notifications until you consolidate.

2. Custom alert rules and recording rules. Update any rules that join on sm_check_info using label_-prefixed names. During dual-write, both forms resolve, so you can migrate these without a gap.

3. Dashboards and queries. Update PromQL and LogQL that references the prefix. During dual-write, you can use or selectors so both forms resolve while you migrate.

4. SLOs. If your SLO queries filter on custom labels, update them to the un-prefixed form.

5. Private probes. Upgrade to the latest agent version before enabling dual-write. The new label behavior is delivered by the agent: older probes keep emitting the old format, leaving you with inconsistent data across probes. Refer to Set up private probes.

6. Checks as code. If you manage checks with Terraform or the API, plan any reserved-label renames in your check definitions. A find-and-replace across your check definitions is usually the fastest path for large fleets.

Migrate

You manage the migration from the Label migration tab in the Synthetic Monitoring app. The tab shows your current migration state, a preview of how the labels on your metrics appear in that state, and the list of reserved system label names.

Before you begin

• You need administrator permissions in Grafana. Without them, the Label migration tab shows the current migration state, but you can’t change it.
• Upgrade your private probes to the latest agent version. Skip this if you only use public probes.
• Before each state change, inform your on-call team and consider silencing Synthetic Monitoring alerts for 10 to 15 minutes.

Enable dual-write

1. Navigate to Testing & synthetics > Synthetics > Config.

2. Click the Label migration tab.

3. Under Current status, click Enable dual-write.

4. Review the confirmation dialog box, then click Enable dual-write to confirm.

   Warning

   This step can’t be undone. After you enable dual-write, you can’t return to prefixed-only labels.

If any of your checks or probes use a custom label that conflicts with a reserved system name, the transition is rejected and the tab lists the conflicting label names. Rename or remove those labels from your checks and probes, then try again. For the full list of reserved names, refer to System labels.

After dual-write is enabled, sm_check_info and your check logs carry both label forms, and your check metrics carry the un-prefixed labels. Expect the brief staleness gap described in What to expect when you migrate: checks repopulate on their next execution. Note that dual-write temporarily doubles the number of custom labels on sm_check_info and your log streams. You can use the How your labels appear right now section of the tab to verify that both label forms are present.

Update your configuration

While in dual-write, audit and update the items in the migration checklist: notification policies, alert and recording rules, dashboards, and SLOs. Then validate that alerts route to the right teams, dashboards render, and SLOs are evaluating.

Finalize the migration

1. Navigate to Testing & synthetics > Synthetics > Config.
2. Click the Label migration tab.
3. Under Current status, click Finalize migration.
4. Review the confirmation dialog box, then click Finalize to confirm.

Only sm_check_info and check logs change: the prefixed label copies are removed. Your check metrics keep the same label set.

If something unexpected occurs, for example an alert routing policy you missed still depends on the prefixed form, click Revert to dual-write on the same tab. This temporarily restores the prefixed form alongside the un-prefixed form while you finish updating your configuration, and you can finalize again at any time.

What happens if you don’t migrate

After the migration deadline, remaining tenants are automatically migrated to the un-prefixed state. If you haven’t updated your alert routing, dashboards, and rules by then, anything that references label_-prefixed names stops matching new data at that point.

Additionally, following the release of the Synthetic Monitoring agent v1.0, older probe versions that emit the prefixed format are deprecated.

FAQ

Will this increase my cardinality or cost?

No. Labels are added to your existing series, so the number of series doesn’t change; each series just carries more labels. At each state change, replaced series remain counted as active for about five minutes. Grafana Cloud billing forgives short-lived spikes through 95th percentile billing, so there is no billing impact.

Why do my checks look empty after enabling dual-write?

Adding labels changes each series’ identity, so the previous series stop receiving data. This is expected: your checks repopulate as new executions occur, typically within one check interval.

Can I still access my historical data?

Yes. Nothing is deleted. Data recorded before the change remains queryable under the old label names for your full retention period. During dual-write, you can use or selectors to query across both forms in one graph.

What happens if Synthetic Monitoring reserves a new label I’m already using?

If Synthetic Monitoring reserves a new label, your label keeps working: Synthetic Monitoring doesn’t overwrite or drop your data. The trade-off is that any system functionality tied to the new label doesn’t apply to your affected checks, and the API blocks edits to those checks until you rename the label. In-app warnings identify affected checks. For details, refer to System labels.

I used to rely on the region label. What happened to it?

The probe-region label is renamed to sm_region so that region is available as a custom label. If you consume the system region label in dashboards, queries, or routing, update those references to sm_region.

Do my logs change too?

Yes. The label_ prefix is removed from log labels as well. Note that Loki indexes a maximum of 15 labels per stream, so during dual-write not all label copies may be indexed if you use many custom labels. You can always filter on any label using LogQL label-filter syntax:

logql Copy

{source="synthetic-monitoring-agent"} | team="checkout"

A note on LBAC for logs: Label-based access control on Synthetic Monitoring log labels is not an officially supported configuration, but some customers use it. LBAC rules that match label_-prefixed log labels stop matching once the prefix is removed. If you rely on this pattern, update your LBAC rules to the un-prefixed form during dual-write. Be aware that Loki indexes at most 15 labels per stream, so for checks with many custom labels the un-prefixed copy may not be indexed during dual-write. Verify that your rules match before finalizing.