Cardinality

Cardinality refers to the total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate. For more information on cardinality, see the What are cardinality spikes and why do they matter? blog post.

Because writes to a time-series database (TSDB) database are in series, high cardinality does not make a big difference to performance at ingest. However, cardinality can have a major impact on querying where, the higher the cardinality, the more items are required to be iterated over.

Traces collection and metrics

Tempo’s server-side metrics generation adds functionality to the collection of traces by creating Prometheus-based metrics that track a variety of metrics such as:

Total span call counts
Span latency histograms
Total span size count

The metrics-generator creates metrics which define the relationship between services via edges and nodes. Each of these metrics are queryable using a set of Prometheus labels (key/value pairs).

Each new value for a label increases the number of active series associated with a metric. (To learn more about active series, read the Trace active series documentation.)

This is also known as an increase in cardinality, and the number of active series generated for a metric is directly proportional to the number of labels that exist for that metrics alongside the number of values each label has added.

In a non-modified instance of the metrics generator, a small number of labels are added automatically. Because labels like span_kind and status_code only have a few valid values, the largest variable for the number of active series produced for each metric depends on the number of service names and span names associated with trace spans.

The metrics-generator can also be configured to also add extra labels on metrics, using span attribute key/value pairs which are mapped directly to these labels see the custom span attribute documentation.

Be careful when configuring custom attributes: the greater the number of values seen in a specific attribute, the greater the number of active series will be produced. For more information about active series, refer to the active series documentation

Let’s say that you are adding a custom attribute that includes unique customer IDs as a metrics label. If you have 100 customers, this could potentially multiply the number of active series generated by up to 100 (for example, going from 25,000 active series to 2.5M). Always consider which attributes will actually be useful as labels for querying metrics, as well as the cardinality that they will increase metrics by.

Dry-running the metrics-generator

An often most reliable solution is by running the metrics-generator in a dry-run mode. Using the dry-run mode generates metrics but does not collecting them, thus not writing them to a metrics storage. The override metrics_generator.disable_collection is defined for this use-case.

To get an estimate, run the metrics-generator normally and set the override to true. Then, check tempo_metrics_generator_registry_active_series to get an estimation of the active series for that set-up.