Label-based access control can be used to create access policies that will only allow for data to be queried that meets specific label requirements. The feature allows multiple sets of Prometheus label selectors to be associated with a policy and queries will only return data from series that match at least one of the provided selectors. This correlates to disjunctive normal form which allows any required policy to be expressed.
Setting up a label policy
Label policies are set when creating an access policy on a per tenant basis. This means each tenant associated with an access policy can have a unique label policy.
Alertmanager and ruler
Label policies are not enforced by the alertmanager and ruler. This means that the requests that they serve contain everything for a particular tenant without applying label-based access control. For example, listing all rule groups in the ruler will return all rule groups for the tenant even if a label selector in the access policy excludes some of the labels on the rules.
In the case of the ruler this only applies to the HTTP endpoints. All metrics that the ruler generates for
alerting or recording rules (alerting rules generate the
ALERTS_FOR_STATE metrics) are subject to
label-based access control when queried.
GEM does not enforce label-based access control on the write requests. This means a
scope in the access policy allows clients to push any metrics without restrictions regarding the labels on the metrics.
Exclude a label
One common use case for creating an LBAC policy is to exclude metrics with a specific label. For instance, a label policy that excludes all series with the label
secret=true would be created by just adding a select with
secret!="true" when creating an access policy. This can be seen in the image below:
Exclude a metric
Expanding upon the previous example, lets say we wanted to create an access policy that only excludes metrics with the label
secret=true on the metric named
sensitive_requests_total. Since the name of a metric is actually just a label with the key
__name__, we can leverage the existing LBAC label selector syntax to enforce this:
You may notice above that two different selectors where added to enforce the policy. Specifically:
The first selector will match when returning a series from the metrics
sensitive_requests_total and will ensure all of the returned series do not have the
secret: true label. However, when requesting a metric besides
sensitive_requests_total, the second label selector will match and return any data even if it has the
secret: true label.
Related Enterprise Metrics resources
Running Prometheus-as-a-service with Grafana Enterprise Metrics
Introducing Grafana Enterprise Metrics (GEM), a simple and scalable Prometheus service that is seamless to use, simple to maintain, and supported by Grafana Labs.
How Robinhood scaled from 100M to 700M time series with Grafana Enterprise Metrics
In this GrafanaCONline session, the Robinhood team tells how GME (GameStop) led to GEM (Grafana Enterprise Metrics).
Benchmarking Grafana Enterprise Metrics for horizontally scaling Prometheus up to 500 million active series
We stress-tested GEM to show how it horizontally scaled. One takeaway: Hardware usage scales linearly up to 500 million active series.