This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Metrics from traces Service graphs

Service graphs

A service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components:

• Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs help you to understand the structure of the system.
• Provide a high-level overview of the health of your system. Service graphs display error rates, latencies, as well as other relevant data.
• Provide an historic view of a system’s topology. Distributed systems change very frequently, and service graphs offer a way of seeing how these systems have evolved over time.

Service graphs can be generated from metrics created by the metrics-generator or Grafana Alloy. Refer to Enable service graphs for more information on how to enable service graphs in Tempo.

How they work

The metrics-generator and Grafana Alloy both process traces and generate service graphs in the form of Prometheus metrics.

Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request. The processor uses the OpenTelemetry semantic conventions to detect a myriad of requests.

It supports the following requests:

• A direct request between two services where the outgoing and the incoming span must have span.kind, client, and server, respectively.
• A request across a messaging system where the outgoing and the incoming span must have span.kind, producer, and consumer respectively.
• A database request; in this case the processor looks for spans containing attributes span.kind=client as well as one of db.namespace, db.name or db.system. See below for how the name of the node is determined for a database request.

The processor keeps every span that can form a request pair in an in-memory store until the corresponding pair span arrives or the maximum waiting time passes. When either condition occurs, the processor records the request and removes it from the local store.

Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.

Copy

  traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20

Virtual nodes

Virtual nodes are nodes that form part of the lifecycle of a trace, but spans for them aren’t collected because they’re outside the user’s reach or aren’t instrumented. For example, you might not collect spans for an external service for payment processing that’s outside user interaction.

The processor detects virtual nodes in two ways:

• Uninstrumented client (missing client span): The root span has span.kind set to server or consumer, with no matching client span. This indicates that the request or message was initiated by an external system that isn’t instrumented, like a scheduler, a frontend application, or an engineer using curl.
  • In the Tempo metrics-generator, the processor checks the configured peer_attributes on the server span first. If it finds a matching attribute, it uses that value as the client node name. Otherwise, the client node name defaults to user.
  • In Grafana Alloy and the OpenTelemetry Collector servicegraph connector, the connector doesn’t evaluate peer attributes for this case. The client node name always defaults to user and you can’t override it. An upstream feature request exists to add this capability.
• Uninstrumented server (missing server span): A client span doesn’t have its matching server span, but has a peer attribute present. In this case, the client called an external service that doesn’t send spans. The processor uses the peer attribute value as the virtual server node name.
  • The default peer attributes are peer.service, db.name, and db.system.
  • The processor searches the attributes in order and uses the first match as the virtual node name.

The processor identifies a database node when the span has at least one db.namespace, db.name, or db.system attribute.

The processor determines the database node name using the following span attributes in order of precedence: peer.service, server.address, network.peer.address:network.peer.port, db.namespace, db.name.

Metrics

The following metrics are exported:

The processor measures duration from both the client and server sides.

Possible values for connection_type: unset, virtual_node, messaging_system, or database.

You can include additional labels using the dimensions configuration option or the enable_virtual_node_label option.

Since the service graph processor has to process both sides of an edge, it needs to process all spans of a trace to function properly. If spans of a trace spread across multiple instances, the processor can’t pair them reliably.

Activate enable_virtual_node_label

Activating this feature adds the following label and corresponding values: