Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Agent Flow mode Reference Components reference otelcol.connector.spanmetrics

otelcol.connector.spanmetrics

EXPERIMENTAL: This is an experimental component. Experimental components are subject to frequent breaking changes, and may be removed with no equivalent replacement.

otelcol.connector.spanmetrics accepts span data from other otelcol components and aggregates Request, Error and Duration (R.E.D) OpenTelemetry metrics from the spans:

• Request counts are computed as the number of spans seen per unique set of dimensions, including Errors. Multiple metrics can be aggregated if, for instance, a user wishes to view call counts just on service.name and span.name.

  Requests are tracked using a calls metric with a status.code datapoint attribute set to Ok:

  Copy

  calls { service.name="shipping", span.name="get_shipping/{shippingId}", span.kind="SERVER", status.code="Ok" }

• Error counts are computed from the number of spans with an Error status code.

  Errors are tracked using a calls metric with a status.code datapoint attribute set to Error:

  Copy

  calls { service.name="shipping", span.name="get_shipping/{shippingId}, span.kind="SERVER", status.code="Error" }

• Duration is computed from the difference between the span start and end times and inserted into the relevant duration histogram time bucket for each unique set dimensions.

  Span durations are tracked using a duration histogram metric:

  Copy

  duration { service.name="shipping", span.name="get_shipping/{shippingId}", span.kind="SERVER", status.code="Ok" }

NOTE: otelcol.connector.spanmetrics is a wrapper over the upstream OpenTelemetry Collector spanmetrics connector. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

Multiple otelcol.connector.spanmetrics components can be specified by giving them different labels.

Usage

river Copy

otelcol.connector.spanmetrics "LABEL" {
  histogram {
    ...
  }

  output {
    metrics = [...]
  }
}

Arguments

otelcol.connector.spanmetrics supports the following arguments:

Adjusting dimensions_cache_size can improve the Agent process’ memory usage.

The supported values for aggregation_temporality are:

• "CUMULATIVE": The metrics will not be reset after they are flushed.
• "DELTA": The metrics will be reset after they are flushed.

If namespace is set, the generated metric name will be added a namespace. prefix.

resource_metrics_cache_size is mostly relevant for cumulative temporality. It helps avoid issues with increasing memory and with incorrect metric timestamp resets.

resource_metrics_key_attributes can be used to avoid situations where resource attributes may change across service restarts, causing metric counters to break (and duplicate). A resource does not need to have all of the attributes. The list must include enough attributes to properly identify unique resources or risk aggregating data from more than one service and span. For example, ["service.name", "telemetry.sdk.language", "telemetry.sdk.name"].

Blocks

The following blocks are supported inside the definition of otelcol.connector.spanmetrics:

It is necessary to specify either a “exponential” or an “explicit” block:

• Specifying both an “exponential” and an “explicit” block is not allowed.
• Specifying neither an “exponential” nor an “explicit” block is not allowed.

dimension block

The dimension block configures dimensions to be added in addition to the default ones.

The default dimensions are:

• service.name
• span.name
• span.kind
• status.code

The default dimensions are always added. If no additional dimensions are specified, only the default ones will be added.

The following attributes are supported:

otelcol.connector.spanmetrics will look for the name attribute in the span’s collection of attributes. If it is not found, the resource attributes will be checked.

If the attribute is missing in both the span and resource attributes:

• If default is not set, the dimension will be omitted.
• If default is set, the dimension will be added and its value will be set to the value of default.

events block

The events block configures the events metric, which tracks span events.

The following attributes are supported:

At least one dimension block is required if enabled is set to true.

histogram block

The histogram block configures the histogram derived from spans’ durations.

The following attributes are supported:

The supported values for unit are:

• "ms": milliseconds
• "s": seconds

exponential block

The exponential block configures a histogram with exponential buckets.

The following attributes are supported:

explicit block

The explicit block configures a histogram with explicit buckets.

The following attributes are supported:

exemplars block

The exemplars block configures how to attach exemplars to histograms.

The following attributes are supported:

max_per_data_point can help with reducing memory consumption.

output block

The output block configures a set of components to forward resulting telemetry data to.

The following arguments are supported:

You must specify the output block, but all its arguments are optional. By default, telemetry data is dropped. Configure the metrics argument accordingly to send telemetry data to other components.

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer traces telemetry data. It does not accept metrics and logs.

Handling of resource attributes

otelcol.connector.spanmetrics is an OTLP-native component. As such, it aims to preserve the resource attributes of spans.

1. For example, let’s assume that there are two incoming resources spans with the same service.name and k8s.pod.name resource attributes.

   Example JSON of two incoming spans.

2. otelcol.connector.spanmetrics will preserve the incoming service.name and k8s.pod.name resource attributes by attaching them to the output metrics resource. Only one metric resource will be created, because both span resources have identical resource attributes.

   Example JSON of one outgoing metric resource.

3. Now assume that otelcol.connector.spanmetrics receives two incoming resource spans, each with a different value for the k8s.pod.name recourse attribute.

   Example JSON of two incoming spans.

4. To preserve the values of all resource attributes, otelcol.connector.spanmetrics will produce two resource metrics. Each resource metric will have a different value for the k8s.pod.name recourse attribute. This way none of the resource attributes will be lost during the generation of metrics.

   Example JSON of two outgoing metric resources.

Component health

otelcol.connector.spanmetrics is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.connector.spanmetrics does not expose any component-specific debug information.

Examples

Explicit histogram and extra dimensions

In the example below, http.status_code and http.method are additional dimensions on top of:

• service.name
• span.name
• span.kind
• status.code

river Copy

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    traces  = [otelcol.connector.spanmetrics.default.input]
  }
}

otelcol.connector.spanmetrics "default" {
  // Since a default is not provided, the http.status_code dimension will be omitted
  // if the span does not contain http.status_code.
  dimension {
    name = "http.status_code"
  }

  // If the span is missing http.method, the connector will insert
  // the http.method dimension with value 'GET'.
  dimension {
    name = "http.method"
    default = "GET"
  }

  dimensions_cache_size = 333

  aggregation_temporality = "DELTA"

  histogram {
    unit = "s"
    explicit {
      buckets = ["333ms", "777s", "999h"]
    }
  }

  // The period on which all metrics (whose dimension keys remain in cache) will be emitted.
  metrics_flush_interval = "33s"

  namespace = "test.namespace"

  output {
    metrics = [otelcol.exporter.otlp.production.input]
  }
}

otelcol.exporter.otlp "production" {
  client {
    endpoint = env("OTLP_SERVER_ENDPOINT")
  }
}

Sending metrics via a Prometheus remote write

The generated metrics can be sent to a Prometheus-compatible database such as Grafana Mimir. However, extra steps are required in order to make sure all metric samples are received. This is because otelcol.connector.spanmetrics aims to preserve resource attributes in the metrics which it outputs.

Unfortunately, the Prometheus data model has no notion of resource attributes. This means that if otelcol.connector.spanmetrics outputs metrics with identical metric attributes, but different resource attributes, otelcol.exporter.prometheus will convert the metrics into the same metric series. This problem can be solved by doing either of the following:

• Recommended approach: Prior to otelcol.connector.spanmetrics, remove all resource attributes from the incoming spans which are not needed by otelcol.connector.spanmetrics.

  Example River configuration to remove unnecessary resource attributes.

• Or, after otelcol.connector.spanmetrics, copy each of the resource attributes as a metric datapoint attribute. This has the advantage that the resource attributes will be visible as metric labels. However, the cardinality of the metrics may be much higher, which could increase the cost of storing and querying them. The example below uses the merge_maps OTTL function.

  Example River configuration to add all resource attributes as metric datapoint attributes.

If the resource attributes are not treated in either of the ways described above, an error such as this one could be logged by prometheus.remote_write: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp).

Compatible components

otelcol.connector.spanmetrics can accept arguments from the following components:

• Components that export OpenTelemetry otelcol.Consumer

otelcol.connector.spanmetrics has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer