Documentation Grafana Cloud Platform Knowledge graph Configure Retained labels

Retain organization labels in the knowledge graph

The knowledge graph normalizes your telemetry into a fixed set of well-known labels (service, namespace, workload, cluster, and so on). Organization-specific labels—such as team, zone, region, or environment—are typically dropped during this normalization, so they don’t reach entities in the knowledge graph or its recording rules in Mimir.

With retained labels you can specify a small set of your own labels to keep. After you configure them, the knowledge graph:

• Attaches each retained label as a property on any entity discovered from a metric that carries the label. You can then filter, group, and drill down on those properties in the entity catalog and workbench.
• Adds each retained label to the knowledge graph’s recording rules, metrics, and insights when doing so doesn’t inflate the tenant’s Mimir series count.

Before you begin

• The customLabelRetention feature flag must be enabled on your Grafana Cloud stack. This is a private-preview flag—ask your Grafana Labs representative to enable it.
• The labels you plan to retain must already exist on your Prometheus metrics. Retained labels preserve labels that are already present on the source series; they don’t invent new ones.

How it works

Retained labels take effect on two sides, at two different speeds.

Entity model (fast)

After you save a retained label, entities in the knowledge graph start carrying it as a property within minutes.

Mimir recording rules (slower)

The knowledge graph also updates its own recording rules to keep the retained label. This propagates to your Mimir over time and can take a few days to fully take effect across all rules.

Configure retained labels

1. Open the knowledge graph app and select Configuration.
2. Select the Retained labels tab.
3. Pick a label from the dropdown. The dropdown lists label keys present on your metrics data source. Selecting a label saves it immediately—there is no separate Save button.
4. To remove a retained label, select the × on its chip.

After you add labels, they appear as removable chips above the dropdown. Newly added labels start applying on the next model-building cycle; recording-rule sync completes a few minutes later.

Validation

Each label is checked for cardinality before it’s persisted. The knowledge graph probes your Prometheus against a small set of representative aggregations and rejects any label that would cause excessive series growth.

Rejected labels aren’t saved and the reason is shown on the label chip; valid labels in the same submission are still saved.

Understand limitations during private preview

• The feature is off by default. Grafana Labs must enable it on your stack before the Retained labels tab appears in the app.
• Individual recording rules may skip a retained label when adding it would inflate the tenant’s Mimir series count. This is deliberate and can vary by tenant and by rule.