Promote resource attributes

Grafana Cloud

Promote OpenTelemetry resource attributes to metric labels

OpenTelemetry metrics carry two kinds of attributes: metric attributes, which describe an individual measurement, and resource attributes, which describe the entity that produced the measurement, such as a service, a container, or a Kubernetes Pod.

When Grafana Cloud Metrics ingests OTLP metrics, it stores metric attributes as labels on the metric series and stores resource attributes in a separate target_info series. Grafana Cloud also writes a configurable subset of the resource attributes, called the promoted attributes, as labels directly on every metric series from that resource. Promoting an attribute means you can filter, group, and join on it without joining against target_info.

How attribute promotion works

Grafana Cloud promotes a set of common attributes by default. Without explicit configuration, these attributes are available as Prometheus labels. To customize promotion, Grafana Cloud provides two parameters that extend or reduce this default set: promote and neverPromote.

promote is a set of attributes that should be promoted in addition to the default set (extend).
neverPromote is a set of attributes that should not be promoted (reduce).

Grafana Cloud computes the promoted set using its defaults and your configuration:

promoted attributes = (defaults + promote) - neverPromote

The two sets promote and neverPromote must be mutually exclusive, and neither list can contain duplicates. Grafana Cloud rejects a configuration that lists the same attribute in both lists, or the same attribute twice in one list.

Use the OpenTelemetry attribute name, with periods (.), in your configuration. Grafana Cloud applies the usual Prometheus name conversion when it writes the label, so the resource attribute k8s.namespace.name becomes the label k8s_namespace_name, and the attribute test.special becomes the label test_special. For more information about name conversion, refer to Metric and label name conversion.

Attributes that aren’t promoted remain available. Grafana Cloud stores all resource attributes in the target_info series, which you can join to your metrics with on(job, instance) or with the Prometheus info() function. Regardless of promotion, Grafana Cloud maps the service.name, service.namespace, and service.instance.id attributes to the job and instance labels. For more information, refer to Resource attributes added to target_info metric.

Before you begin

Caution
Self-service configuration for this feature is still experimental. If you would like to configure your resource attributes promotion, contact Grafana Labs Support.

Before you make changes to the promoted set of resource attributes, consider the following:

Promotion affects cardinality and cost. Each promoted attribute adds a label to every series produced by the resource, and each distinct value of that attribute creates a separate set of series. Promote attributes with a small, stable set of values, such as a team, a tier, or a region. Avoid attributes that are unique per process, container, or request, such as container.id, host.id, or process.pid. For more information about how series count affects your bill, refer to Metrics pricing.
Changes apply to newly ingested data. Existing series keep the labels they were ingested with. Adding a promoted attribute starts a new series for each affected metric, and removing one does the same, so queries that span the change see a discontinuity. Plan for that in dashboards, recording rules, and alert rules that group by the affected labels.
Promotion applies to metrics only. Grafana Cloud Logs promotes its own set of resource attributes for OTLP logs, which you configure separately. For more information, refer to Configure Loki settings.

Getting started

Familiarize yourself with how Grafana Cloud converts OTLP metrics, and with the metrics ingestion limits that apply to resource attributes. For more information, refer to OTLP format considerations.
Verify that you have access to the Grafana instance with Organization Admin privileges. For more information, refer to About users and permissions.
Create a temporary service account with Admin rights, and add a token for that service account to be used with the gcx command line tool.
The gcx the command line tool is installed and set up to interact with your Grafana Cloud resources. For instructions on installation, configuration, and usage, refer to Introduction to gcx.

Note
While we use gcx, the older grafanactl CLI accepts the same commands shown in the examples below. Since grafanactl is no longer actively developed, prefer gcx.

The MimirOTLPConfiguration resource

Promoted attributes are managed through a declarative workflow. You define the configuration as a MimirOTLPConfiguration Kubernetes Custom Resource and push it with gcx. After submission, Grafana Cloud validates the configuration and applies it to the Mimir environment associated with your stack.

The following example promotes the test.special attribute in addition to the defaults, and stops promoting the never.promoted attribute:

apiVersion: mimir.ext.grafana.com/v1alpha1
kind: MimirOTLPConfiguration
metadata:
  name: config
spec:
  resourceAttributes:
    promote:
      - test.special
    neverPromote:
      - never.promoted

The resource uses the following fields:

apiVersion: Specifies the API version of the resource, and is currently:
YAML
```
apiVersion: mimir.ext.grafana.com/v1alpha1
```
kind: Specifies the type of Kubernetes resource, and must be:
YAML
```
kind: MimirOTLPConfiguration
```
metadata.name: Defines a unique name for the resource within the namespace related to your stack.
spec.resourceAttributes.promote: A list of OpenTelemetry resource attribute names to promote in addition to the default set. Resource attributes may not contain leading or trailing spaces or commas (,). All other characters are allowed.
spec.resourceAttributes.neverPromote: A list of OpenTelemetry resource attribute names to exclude from promotion, including attributes from the default set.

Caution
Grafana Cloud features rely on specific resource attributes, and some of them might not work correctly if you stop promoting those attributes. Review which dashboards, recording rules, and alert rules depend on a label before you add its attribute to neverPromote.

View the current configuration

To check whether your stack already has a promotion configuration, complete the following steps:

Select the context for the stack you want to inspect:
Bash
```
gcx config use-context <CONTEXT>
```
For more details on how to correctly configure contexts, refer to the gcx configuration.
List the MimirOTLPConfiguration resources in the stack:
Bash
```
gcx resources get mimirotlpconfigurations
```
If no output is returned, no configuration exists and your stack promotes the default set of attributes. Refer to Create a configuration.
If you see output similar to the following, a configuration already exists. Refer to Update an existing configuration.
text
```
KIND                      GROUP                   NAME
MimirOTLPConfiguration    mimir.ext.grafana.com   mimir-otlp-config
```
Use the name from the NAME column in the commands that follow. The examples in this documentation assume config.

Create a configuration

To promote attributes for a stack that doesn’t have a configuration yet, complete the following steps:

Select the context for the stack:
Bash
```
gcx config use-context <CONTEXT>
```
Create a file named config.yaml, for example at /path/to/otlp-config/config.yaml, that defines a MimirOTLPConfiguration resource.
The following example promotes two additional attributes, team and cloud.provider, and keeps the entire default set:
YAML
```
apiVersion: mimir.ext.grafana.com/v1alpha1
kind: MimirOTLPConfiguration
metadata:
  name: config
spec:
  resourceAttributes:
    promote:
      - team
      - cloud.provider
```
Push the configuration:
Bash
```
gcx resources push --path /path/to/otlp-config/config.yaml
```
For more details on how to manage resources using gcx, refer to Manage resources with gcx.

Note
A stack may contain at most one MimirOTLPConfiguration resource. Creating a second one fails even if you give it a different name.

Update an existing configuration

To change the promoted attributes for a stack that already has a configuration, edit the existing resource instead of writing a new one from scratch. A push replaces the whole spec, so an attribute you omit from promote is no longer promoted.

Pull the existing resource:

gcx resources pull mimirotlpconfiguration/config --path <PATH-TO-LOCAL-COPY> -o yaml

The command stores the resource locally at:

<PATH-TO-LOCAL-COPY>/mimirotlpconfigurations.v1alpha1.mimir.ext.grafana.com/config.yaml

Create a backup of the pulled file in a separate folder, so that you can restore the previous configuration if needed.

Edit <PATH-TO-LOCAL-COPY>/mimirotlpconfigurations.v1alpha1.mimir.ext.grafana.com/config.yaml and update the spec.resourceAttributes section. For example, the updated configuration can look like this:

apiVersion: mimir.ext.grafana.com/v1alpha1
kind: MimirOTLPConfiguration
metadata:
  name: config
spec:
  resourceAttributes:
    promote:
      - team
      - cloud.provider
      - cloud.platform # this is a newly promoted attribute
    neverPromote:
      - k8s.pod.name # this default attribute is no longer promoted

Push the updated configuration:

gcx resources push mimirotlpconfiguration/config --path <PATH-TO-LOCAL-COPY>/mimirotlpconfigurations.v1alpha1.mimir.ext.grafana.com/config.yaml

Revert to the default set

To return your stack to the default set of promoted attributes, delete the resource. Removing the spec.resourceAttributes section and pushing the resource has the same effect on the promoted set.

Select the context for the stack:
Bash
```
gcx config use-context <CONTEXT>
```

Delete the resource:

gcx resources delete mimirotlpconfiguration/config

After deletion, no promotion overrides remain for the stack. If you need them again, define them from scratch.

Verify the configuration

To see whether Grafana Cloud has applied the configuration, pull the resource and inspect its status section:

gcx resources pull mimirotlpconfiguration/config --path <PATH-TO-LOCAL-COPY> -o yaml

The mimir-config-controller entry reports the outcome of the last reconciliation:

status:
  operatorStates:
    mimir-config-controller:
      state: success
      descriptiveState: 'Last synchronized object version: ...'
      lastEvaluation: ...

The state field is either success or failed. If the state is failed, descriptiveState contains the error.

OTLP: OpenTelemetry Protocol format considerations
Send data to the Grafana Cloud OTLP endpoint
Resources in the OpenTelemetry documentation

Was this page helpful?

Email docs@grafana.com

Help and support

Community

Promote OpenTelemetry resource attributes to metric labels

How attribute promotion works

Before you begin

Getting started

The MimirOTLPConfiguration resource

View the current configuration

Create a configuration

Update an existing configuration

Revert to the default set

Verify the configuration

Was this page helpful?

Still have questions?

Get every update

Promote OpenTelemetry resource attributes to metric labels

How attribute promotion works

Before you begin

Getting started

The MimirOTLPConfiguration resource

View the current configuration

Create a configuration

Update an existing configuration

Revert to the default set

Verify the configuration

Related topics

Was this page helpful?

Related resources from Grafana Labs