Menu
Grafana Cloud Instrument and send data Traces Metrics-generator in Grafana Cloud Traces
Grafana Cloud

Metrics-generator in Grafana Cloud Traces

The Tempo metrics-generator can derive metrics from traces as they are ingested. When used in Grafana Cloud, the metrics-generator writes metrics directly to the hosted Prometheus instance in the same stack.

Metrics-generator in Grafana Cloud Traces architecture.
Metrics-generator in Grafana Cloud Traces architecture.

For more information about the metrics-generator and the metrics it creates, see Grafana Tempo | Metrics-generator. This document describes the Grafana Cloud-specific capabilities.

Note: Metrics generation is disabled by default. Contact Grafana Support to enable metrics generation for your organization!

Constraints and good to know

  • The active series sent to the hosted Prometheus instance is billed like regular metrics.
  • Metrics can only be sent to a hosted Prometheus instance in the same region.
  • If traces are down-sampled before reaching Tempo, the metrics will be lower than reality.

Monitoring the metrics-generator

The grafanacloud-usage data source exposes several metrics about the metrics-generator.

Amount of active series:

grafanacloud_traces_instance_metrics_generator_active_series{}

Amount of active series being limited:

grafanacloud_traces_instance_metrics_generator_series_dropped_per_second{}

Amount of spans that are discarded by the metrics-generator before they are processed:

grafanacloud_traces_instance_metrics_generator_discarded_spans_per_second

This metric has a reason label:

  • outside_metrics_ingestion_slack: The time between the creation of the span and when it was ingested was too large and the span is deemed outdated. Processing this span and including it a current metrics sample would skew the data.

How this works

When the amount of active series in Tempo reaches a configurable limit, no new active series are added. Grafana Cloud Traces keeps updating the existing series. The series exceeding the limit are dropped.

Configuration options

The following settings can be configured in Grafana Cloud Traces. Contact Grafana Support to tweak any of these settings.

ConfigurationDescription
Enabled processorThe metrics processors to enable; options include service graphs and/or span metrics.
Max active seriesThe maximum amount of active series.
Collection intervalHow often samples are collected from the active series. Defaults to every 15s or 4 DPM.
Histogram bucketsThe buckets used for the histograms generated by the metrics-generator. This can be configured per processor.
DimensionsAdditional dimensions to be added to the generated metrics. If this dimension is present in the span attributes, it is included as a label in the metrics. This can be configured per processor.
Note: Characters that are not valid Prometheus labels are sanitized. For example the trace attribute k8s.namespace becomes the Prometheus label k8s_namespace.