This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.
Monitor Tempo instances and the operator
You can configure the Tempo Operator to monitor TempoStack instances (including all Tempo components like the distributor). In addition, the operator can expose metrics about the operator itself (for example, the number of successful and failed upgrades, etc.).
Monitor TempoStack instances
The Tempo Operator supports monitoring and alerting of each Tempo component (distributor, ingester, etc.).
To enable metrics and alerting, the Prometheus Operator or a comparable solution which discovers ServiceMonitor
and PrometheusRule
objects must be installed and configured in the cluster.
The configuration for monitoring TempoStack
instances is exposed in the CR:
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
spec:
observability:
metrics:
createServiceMonitors: true
createPrometheusRules: true
tracing:
sampling_fraction: 1.0
jaeger_agent_endpoint: localhost:6831
Configure distributed tracing of operands
All Tempo components as well as the Tempo Gateway support the export of traces in thrift_compact
format.
Deploy OpenTelemetry collector sidecar
To deploy the OpenTelemetry collector, follow these steps:
- Install OpenTelemetry Operator into the cluster.
- Create an
OpenTelemetryCollector
CR that receives trace data in Jaeger Thrift format and exports data via OTLP to the desired trace backend. - Optional: Deploy tracing backend to store trace data.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: sidecar-for-tempo
spec:
mode: sidecar
config: |
receivers:
jaeger:
protocols:
thrift_compact:
exporters:
otlp:
endpoint: <otlp-endpoint>:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [jaeger]
exporters: [otlp]
Send trace data to OpenTelemetry sidecar
Finally, create a TempoStack
instance that sets jaeger_agent_endpoint
to report trace data to the localhost
.
The Tempo operator sets the OpenTelemetry inject annotation sidecar.opentelemetry.io/inject": "true
to all TempoStack
pods.
The OpenTelemetry Operator will recognize the annotation, and it will inject a sidecar into all TempoStack
pods.
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: simple-stack
spec:
template:
queryFrontend:
jaegerQuery:
enabled:
storage:
secret:
type: s3
name: minio-test
storageSize: 200M
observability:
tracing:
sampling_fraction: "1.0"
jaeger_agent_endpoint: localhost:6831
Monitor the operator
The Tempo Operator can expose upgrade and other operational metrics about the operator itself, and can create alerts based on these metrics. For example, the operator handles Tempo upgrades and exposes metrics like “the number of successful Tempo upgrades”, “number of failed Tempo upgrades”, and others. The operator also creates alerts to notify system administrators if any Tempo upgrade fails.
Other metrics are internal to the operator itself, for example, the duration of a reconcile loop iteration. This operator-specific component continuously tries to match the expected state as described in the TempoStack custom resource to the actual cluster state. For example, if an object is deleted in the cluster which is managed by the operator, the operator re-creates this object again, to match the expected state of the cluster.
The operator can be configured using the ConfigMap tempo-operator-manager-config
in the same namespace as the operator.
The following excerpt shows the configuration options to enable the creation of ServiceMonitor
(for scraping metrics) and PrometheusRule
(for creating alerts) objects:
apiVersion: v1
kind: ConfigMap
data:
controller_manager_config.yaml: |
featureGates:
observability:
metrics:
createServiceMonitors: true
createPrometheusRules: true