--- title: "Configure client-side service graph and span metrics | Grafana Cloud documentation" description: "Learn how to configure client-side service graph and span metric generation using Grafana Alloy for the best knowledge graph experience." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Configure client-side service graph and span metrics By default, the knowledge graph relies on [Tempo metrics generation](/docs/grafana-cloud/send-data/traces/configure/metrics-generator/) to derive metrics from traces. You can also generate these metrics on the client side using [Grafana Alloy](/docs/grafana-cloud/send-data/alloy/). This page describes how to configure two types of client-side trace-derived metrics optimized for the knowledge graph experience: - **Service graph metrics**—edges between services, including request rate, error rate, and latency. - **Span metrics**—RED (request, error, duration) metrics derived from individual spans. ## Before you begin Before you begin, ensure you have the following: - Your services are instrumented with OpenTelemetry and sending traces to Alloy. - You have a Prometheus-compatible metrics backend (for example, Grafana Mimir) configured to receive metrics. - Tempo server-side metrics generation is **disabled** to avoid duplicate metrics (refer to [disable server-side metric generation](/docs/grafana-cloud/send-data/traces/configure/metrics-generator/#enable-metrics-generator) for details). ## How it works ### Service graph metrics [otelcol.connector.servicegraph](/docs/grafana-cloud/send-data/alloy/reference/components/otelcol/otelcol.connector.servicegraph) processes spans from your traces and emits metrics that represent the edges in a service graph. Each metric represents a request between two services (`client` and `server`), and includes: - Request rate (`traces_service_graph_request_total`) - Error rate (`traces_service_graph_request_failed_total`) - Request duration from client and server perspectives (`traces_service_graph_request_client`, `traces_service_graph_request_server`) The connector must process both sides of a span pair, so **all spans of a trace must be processed by the same Alloy instance**. If your traces are distributed across multiple Alloy instances, use `otelcol.exporter.loadbalancing` in front of the instances running `otelcol.connector.servicegraph`. This ensures span pairs are routed to the same instance. Refer to [otelcol.connector.servicegraph](/docs/grafana-cloud/send-data/alloy/reference/components/otelcol/otelcol.connector.servicegraph/#otelcolconnectorservicegraph) for details. ### Span metrics [otelcol.connector.spanmetrics](/docs/grafana-cloud/send-data/alloy/reference/components/otelcol/otelcol.connector.spanmetrics) aggregates RED (request, error, duration) metrics from individual spans. This helps if your system doesn’t have dedicated Prometheus metrics but has distributed tracing implemented—you get RED metrics from your tracing pipeline. The connector generates two metrics per span: - A **counter** tracking request totals. - A **histogram** tracking operation duration. Default dimensions include `service.name`, `span.name`, `span.kind`, and `status.code`. Additional dimensions can be added from span or resource attributes. ## Configure using Grafana Alloy Use this method if you manage Alloy directly, without the `k8s-monitoring` Helm chart. ### Prerequisite You have [Grafana Alloy](/docs/alloy/latest/get-started/install/) installed and configured. ### Service graph metrics The following example receives traces over OTLP, generates service graph metrics, and forwards them to Mimir. The original traces are also forwarded to Tempo. Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```alloy otelcol.receiver.otlp "default" { grpc { endpoint = "0.0.0.0:4317" } output { traces = [ otelcol.connector.servicegraph.default.input, otelcol.exporter.otlphttp.grafana_cloud_traces.input, ] } } otelcol.connector.servicegraph "default" { // Recommended dimensions for Knowledge Graph dimensions = [ "namespace", "service.namespace", "k8s.cluster.name", "k8s.namespace.name", "k8s.pod.name", "k8s.deployment.name", "deployment.environment.name" ] db.name = "db.name" output { metrics = [otelcol.exporter.prometheus.default.input] } } otelcol.exporter.prometheus "default" { forward_to = [prometheus.remote_write.mimir.receiver] } prometheus.remote_write "mimir" { endpoint { url = "https:///api/prom/push" basic_auth { username = sys.env("") password = sys.env("") } } } otelcol.exporter.otlphttp "grafana_cloud_traces" { client { endpoint = "https:///otlp" auth = otelcol.auth.basic.grafana_cloud_traces.handler } } otelcol.auth.basic "grafana_cloud_traces" { username = sys.env("") password = sys.env("") } ``` To verify, search for the following metric in **Metrics Drilldown** against your Mimir data source: promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```promql traces_service_graph_request_total ``` ### Span metrics The following example generates span metrics alongside forwarding traces. It adds the recommended labels as extra dimensions, and flushes metrics every 15 seconds. Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```alloy otelcol.receiver.otlp "default" { http {} grpc {} output { traces = [ otelcol.connector.spanmetrics.default.input, otelcol.exporter.otlphttp.grafana_cloud_traces.input, ] } } otelcol.connector.spanmetrics "default" { // The following are added on top of the default dimensions: // service.name, span.name, span.kind, status.code dimension { name = "service.namespace" } dimension { name = "service.version" } dimension { name = "status.message" } dimension { name = "k8s.cluster.name" } dimension { name = "k8s.pod.name" } dimension { name = "k8s.namespace.name" } dimension { name = "deployment.environment.name" } dimension { name = "db.name" } dimension { name = "db.operation" } dimension { name = "db.statement" } dimension { name = "net.peer.name" } dimension { name = "net.peer.port" } dimension { name = "http.status.code" } histogram { unit = "s" exponential {} } metrics_flush_interval = "15s" output { metrics = [otelcol.exporter.prometheus.default.input] } } otelcol.exporter.prometheus "default" { forward_to = [prometheus.remote_write.mimir.receiver] } prometheus.remote_write "mimir" { endpoint { url = "https:///api/prom/push" basic_auth { username = sys.env("") password = sys.env("") } } } otelcol.exporter.otlphttp "grafana_cloud_traces" { client { endpoint = "https:///otlp" auth = otelcol.auth.basic.grafana_cloud_traces.handler } } otelcol.auth.basic "grafana_cloud_traces" { username = sys.env("") password = sys.env("") } ``` To verify, search for the following metric in **Metrics Drilldown** against your Mimir data source: promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```promql traces_spanmetrics_calls_total ``` ## Configure using the Kubernetes Monitoring Helm chart Use this method if you are using the [`grafana/k8s-monitoring`](https://github.com/grafana/k8s-monitoring-helm) Helm chart. ### Prerequisite You have the `grafana/k8s-monitoring` Helm chart deployed with a traces destination configured (for example, Grafana Cloud Tempo). ### Service graph metrics `serviceGraphMetrics` is configured **per destination**, nested under the traces destination in `destinations`. When enabled, the chart automatically deploys a dedicated Alloy `StatefulSet` with a load balancer in front of it, ensuring span-pair locality. Add or update the traces destination in your `values.yaml`: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml destinations: - name: grafana-cloud-traces type: otlp url: protocol: grpc auth: type: basic username: '' password: '' traces: enabled: true metrics: enabled: false logs: enabled: false processors: serviceGraphMetrics: enabled: true # Recommended dimensions for the Knowledge Graph. dimensions: - namespace - service.namespace - k8s.cluster.name - k8s.namespace.name - k8s.pod.name - k8s.deployment.name - deployment.environment.name # Attribute name used to identify the database name from span attributes. databaseNameAttribute: 'db.name' ``` To verify, confirm the StatefulSet is running and search for the following metric in **Metrics Drilldown**: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl get statefulset -n -l app.kubernetes.io/component=service-graph-metrics ``` promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```promql traces_service_graph_request_total ``` ### Span metrics `spanMetrics` is configured under `applicationObservability.connectors`, which is the section that enables the application observability pipeline (OTLP receiver + trace processing). By default, it includes `service.name`, `span.name`, `span.kind`, and `status.code` as dimensions. Add the following to your `values.yaml`: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml applicationObservability: enabled: true receivers: otlp: grpc: enabled: true port: 4317 http: enabled: true port: 4318 connectors: spanMetrics: enabled: true # Dimensions added on top of the default set: # [service.name, span.name, span.kind, status.code] dimensions: - name: 'service.namespace' - name: 'service.version' - name: 'status.message' - name: 'k8s.cluster.name' - name: 'k8s.pod.name' - name: 'k8s.namespace.name' - name: 'deployment.environment.name' - name: 'db.name' - name: 'db.operation' - name: 'db.statement' - name: 'net.peer.name' - name: 'net.peer.port' - name: 'http.status.code' # Dimensions to exclude from the default set. excludeDimensions: [] ``` To verify, search for the following metric in **Metrics Drilldown** against your Mimir data source: promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```promql traces_span_metrics_calls_total ``` ## Configure using auto-instrumentation with Beyla If your services are not yet instrumented with OpenTelemetry, you can use [Beyla](/docs/beyla/latest/) to generate RED metrics and traces without modifying your application code. When deployed with the `grafana/k8s-monitoring` Helm chart, Beyla automatically generates metrics in a format compatible with knowledge graph and Application Observability. You can enable it through the `autoInstrumentation` section of your `values.yaml`: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml autoInstrumentation: enabled: true ``` The easiest way to get started is through the [Kubernetes Monitoring configuration UI](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/), which generates a complete Helm deployment script with the correct settings for your stack. For more info on this topic, refer to [Deploy Beyla with the Kubernetes Monitoring Helm chart](/docs/beyla/latest/setup/kubernetes-helm-k8s-monitoring/). ## Troubleshooting Expand table | Symptom | Likely cause | Resolution | |-------------------------------------------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------| | No service graph or span metrics in Mimir | Misconfigured Alloy pipeline | Verify the `output.metrics` chain from `otelcol.connector.servicegraph` or `otelcol.connector.spanmetrics` to `prometheus.remote_write` | | Duplicate service graph or span metrics | Tempo server-side metrics generation enabled | Disable server-side metrics generation in Tempo | | Missing edges in entity graph | Services without instrumentation not captured | Configure `virtual_node_peer_attributes` |