Configuring custom active series trackers
You can use the custom tracker feature to count the number of active series on an ingester that match a particular label pattern.
The label pattern to match against is specified using the -ingester.active-series-custom-trackers
CLI flag (or its respective YAML configuration option). Each custom tracker is defined as a key-value pair, where the key is the name of the tracker and the value is the label matcher. Both the key and the value are type <string>
.
The following example configures a custom tracker to count the active series coming from dev
and prod
namespaces for each tenant.
active_series_custom_trackers:
dev: '{namespace=~"dev-.*"}'
prod: '{namespace=~"prod-.*"}'
If you configure a custom tracker for an ingester, the ingester exposes a cortex_ingester_active_series_custom_tracker
gauge metric on its /metrics endpoint.
Each custom tracker counts the active series matching its label pattern on a per-tenant basis, which means that each custom tracker generates as many as # of tenants
series with metric name cortex_ingester_active_series_custom_tracker
. To reduce the cardinality of this metric, only custom trackers that have matched at least one series are exposed on the metric, and they are removed if they become 0
.
Series with metric name cortex_ingester_active_series_custom_tracker
have two labels applied: name
and user
. The value of the name
label is the name of the custom tracker specified in the configuration. The value of the user
label is the tenant-id for which the series count applies.
Assume two custom trackers are configured as in the example above, and that your Grafana Mimir cluster has three tenants: tenant_1
, tenant_2
, and tenant_with_only_prod_metrics
. Assume all series within tenant_with_only_prod_metrics
have labels that match the pattern {namespace=~"prod-.*"}
and none that match {namespace=~"dev-.*"}
.
In this example, the following output appears when the /metrics
endpoint for the ingester component is scraped:
cortex_ingester_active_series_custom_tracker{name="dev", user="tenant_1"}
cortex_ingester_active_series_custom_tracker{name="prod", user="tenant_2"}
cortex_ingester_active_series_custom_tracker{name="prod", user="tenant_with_only_prod_metrics"}
Note: The custom active series trackers are exposed on each ingester. To understand the count of active series matching a particular label pattern in your Grafana Mimir cluster at a global level, you must collect and sum this metric across all ingesters. If you’re running Grafana Mimir with a
replication_factor
> 1, you must also adjust for the fact that the same series will be replicatedRF
times across your ingesters.