Service graphs on Grafana Labs

Enable service graphs

Fri, 03 Apr 2026 16:45:36 +0000

Enable service graphs

You can enable service graphs in Tempo by using the metrics-generator or by configuring service graphs in Grafana Alloy.

Using the metrics-generator to generate service graphs is more efficient and recommended for larger installations.

Note
Cardinality can pose a problem when you have lots of services. To learn more about cardinality and how to perform a dry run of the metrics-generator, refer to the Cardinality documentation.

Enable service graphs with metrics-generator

Service graphs are generated in Tempo and pushed to a metrics storage. Then, they can be represented in Grafana as a graph. You need those components to fully use service graphs.

Enable service graphs in Tempo

To enable service graphs in Tempo, enable the metrics-generator and add an overrides section which enables the service-graphs generator. For more information, refer to the configuration details.

To enable service graphs when using Grafana Alloy, refer to the Grafana Alloy and service graphs section.

Enable service graphs in Grafana

Note
Service graphs are enabled by default in Grafana. Prior to Grafana 9.0.4, service graphs were hidden under the feature toggle tempoServiceGraph.

Configure a Tempo data source’s service graphs by linking to the Prometheus backend where metrics are being sent:

apiVersion: 1
datasources:
  # Prometheus backend where metrics are sent
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: <prometheus-url>
    jsonData:
      httpMethod: GET
    version: 1
  - name: Tempo
    type: tempo
    uid: tempo
    url: <tempo-url>
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: 'prometheus'
    version: 1

Enable service graphs using Alloy

A service graph is a visual representation of the interrelationships between various services. Service graphs help to understand the structure of a distributed system, and the connections and dependencies between its components.

The same service graph metrics can also be generated by Tempo. This is more efficient and recommended for larger installations. For a deep look into service graphs, visit this section.

Service graphs are also used in the application performance management dashboard. For more information, refer to the service graph view documentation.

Before you begin

Service graphs are generated in Grafana Alloy and pushed to a Prometheus-compatible backend. Once generated, they can be represented in Grafana as a graph. You need these components to fully use service graphs.

Steps

To start using service graphs, enable the feature in the Alloy configuration.

The following example adds the http.method and http.target span attributes as Prometheus labels to the generated service graph metrics, before writing the metrics to the Grafana OTLP gateway. Received trace spans are immediately written to the OTLP gateway.

otelcol.receiver.otlp "default" {
  grpc {}
  http {}

  output {
    traces = [
      otelcol.connector.servicegraph.default.input,
      otelcol.exporter.otlp.default.input,
    ]
  }
}

otelcol.connector.servicegraph "default" {
  dimensions = ["http.method", "http.target"]
  output {
    metrics = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = env("OTLP_ENDPOINT")
  }
}

To see all the available configuration options, refer to the component reference.

Create custom service graphs

Fri, 03 Apr 2026 16:45:36 +0000

Create custom service graphs

You can craft custom service graphs based on the metrics generated from metrics-generator.

Before you begin

For custom service graphs, you’ll use:

Tempo metrics-generator
Grafana deployment of version v10 or higher
Grafana Node graph panel

The metrics-generator creates multiple metrics, including traces_service_graph_request_total. This metric holds:

relationships between services
total number of requests performed between services

Create one or more Grafana dashboards

In Grafana, create a new dashboard.
Add two variables:
- Data source of type Prometheus.
- Service of type Label values, enable multi-value option

Add a panel

Create a panel with a single query called edges.
Select your Prometheus data source with metrics from the metrics-generator.

Query using the following example:

label_join(
  label_join(
    label_join(
      sum(increase(traces_service_graph_request_total{server=~"$service"}[5m])) by (server, client) > 0
      or
      sum(increase(traces_service_graph_request_total{client=~"$service"}[5m])) by (server, client) > 0,
    "source", "", "client"),
  "target", "", "server"),
"id", "-", "server", "client")

Use Instant query type.
If you encounter issues, switch to the Table data visualization. Refer to the Node graph panel documentation for more options and expected data shape.

All data transformations are done with the Prometheus label_join operators.

Explanation of the query:

The first label_join creates a new field id which is required by the Node graph panel.
The second and third label_join copy the client and server labels to source and target respectively, as these names are also required by Node graph panel.
traces_service_graph_request_total is queried twice with the OR operator to get a combination of requests from and to all of the selected services.

This query gets most of the job done. However, it’s limitations can’t be compensated for even using the Grafana transform data feature. These limitations include:

Unable to add request stats to nodes and edges, such as req/sec and error rates
Unable to add custom icons for nodes

These limitations can be overcome by wrapping the Prometheus query into a REST API, which provides more flexibility with query results data transformations. You can use Grafana Infinity data source to visualize data from REST API.

Service graph view

Fri, 03 Apr 2026 16:45:36 +0000

Service graph view

Grafana’s service graph view uses metrics generated by the metrics-generator or Grafana Alloy to display span request rates, error rates, and durations, as well as service graphs. Once the requirements are set up, this pre-configured view is immediately available.

Using the service graph view, you can:

Discover spans which are consistently erroring and the rates at which they occur
Get an overview of the overall rate of span calls throughout your services
Determine how long the slowest queries in your service take to complete
Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals)

Requirements

You have to enable span metrics and service graph generation on the Grafana backend so metrics that are generated as traces are ingested.

To use the service graph view, you need:

Tempo or Grafana Cloud Traces with either the metrics generator enabled and configured or Grafana Alloy enabled and configured to send data to a Prometheus-compatible metrics store
Services graphs, which are enabled by default in Grafana
Span metrics enabled in your Tempo data source configuration

The service graph view can be derived from metrics generated by either the metrics-generator or Grafana Alloy.

For information on how to configure these features, refer to the Tempo data sources documentation.

What does the service graph view show?

Using this view, you can see the top five spans with a type of server (listed in the Name column). You can refine any of this data using the filters. Selecting any of the data points lets you see more specific data.

The service graph view provides a span metrics visualization (table) and service graph (node map). In addition, you can use the filters to customize the data displayed.

You can select any information in the table that has an underline to show more detailed information. You can also select any node in the service graph to display additional information.

Error rate example

Let’s say you want to learn more about why cortex.Ingester has the highest error rates. Selecting the second row of the Error rate column displays details about the span metrics in a new window on the right side.

The metrics query used to generate the data appears in the Metrics browser field.

Span metrics table

The span metrics, shown in the table, are generated by the metrics-generator or Grafana Alloy. These metrics are created from ingested tracing data, including RED metrics.

Span metrics generate two metrics:

A counter that computes requests
A histogram that tracks the distribution of durations of all requests

For information about span metrics and how they are calculated, refer to the Span metrics documentation.

Table contents

The span metrics table contains seven columns with five column headings. Selecting a heading sorts the data by ascending or descending values.

Column	Explanation	PromQL query for span
Name	Use the span name. OTel semantic conventions generally expect the span name to be some kind of low cardinality indicator of the http route or database function being performed.	N/A
Rate	LCD gauge (horizontal bar graph). Instances per second of the span. Clicking this field can jump to the appropriate metrics.	`sum(rate( traces_spanmetrics_calls_total{ span_name="", <filters> }[$__range]))`
Error Rate	Number and LCD gauge (horizontal bar graph). Clicking this field shows more detailed metrics.	`sum(rate( traces_spanmetrics_calls_total{ span_name="", span_status="STATUS_CODE_ERROR", <filters> }[$__range]))`
Duration	p90 duration: 90% of all occurrences of this span complete within this time. Clicking this field shows the appropriate metrics.	`histogram_quantile(.9, sum(rate( traces_spanmetrics_duration_seconds_bucket{ span_name="", span_status="STATUS_CODE_ERROR", <filters> }[$__range]) by (le))`
Links	Provide links to example traces given the span name and other applied filters. Link to a search for all spans with the same name from the same Tempo data source.	N/A

Service graphs

A service graph (node graph) is a visual representation of the interrelationships between various services. Service graphs help to understand the structure of a distributed system, and the connections and dependencies between its components.

Service graphs infer the topology of a distributed system, provide a high level overview of the health of your system, and a historic view of a system’s topology. Service graphs show error rates and latencies, among other relevant data. The service graph layout can be the default or grid.

If you are using the metrics-generator, then it processes traces and generates service graphs in the form of time series metrics like:

traces_service_graph_request_total{client="app", server="db"} 20

For information about service graphs and how they are calculated, refer to the Service Graphs documentation.

Use filters to reveal details

The service graph view uses service graphs and span metrics to provide a gateway to your tracing information. This dashboard is derived from a fixed set of metrics queries. These underlying queries can not be changed. However, you can choose which traces are included in the metrics query by filtering.

You can explore data by clicking on selectable items or by using filters.

Select items or nodes for more detail

Clicking on selectable items, such as underlined text in the table or nodes on the service graph, lets you reveal specific details based upon your selection.

In the table, you can select items in the Rate, Error Rate, Duration (p90), and Links columns. Choosing one of these items provides details about the span metrics.

You can view request rate, request histogram, failed request rate, and traces for any node in the service graph. To view more information, select the node in the service graph and then choose an option from the popup. For details on navigating the service graph, refer to the Node graph panel documentation.

Filter with metric queries

Using the filters at the top of the screen, you can narrow the data set based upon span attributes (key-value pairs or labels). The filters build a query to refine what is shown in the service graph and span metrics. You can add one or more label filters.

To use the filters:

At the top of the Service Graph, select the text box after Filter to display a list of available labels.
Select or search for a value for the label. In this case, the value of server is equal to tempo-ingester. The default operator is equals (=).
Optional: Change the operator by selecting = and choosing a new option from the drop-down.
Optional: Add additional key-value pairs to refine the data set. Any subsequent label filters use AND, which requires both key-value pairs to be presents for matches.
Select Run query.

Filters can be removed by selecting the filter drop-down and choosing – remove filter –.

Each field or label represents a key-value pair. Number 1 selects a service as the label whose value is Go-http-client (2). The second key-value pair has a client as a label whose value is 02e807.

If your metrics queries are too specific, they may not return any results.

Updating the filter to be less specific returns a result. In this case, the results show only span metrics data associated with the span_name label with a value of /base.Ruler/Rules. No service graph data was available.

Analyze service graph data

Fri, 03 Apr 2026 16:45:36 +0000

Analyze service graph data

You can use this collection of PromQL queries to explore service graph metrics.

Grafana uses the service graph metrics created by Tempo and builds that visual for the user. However, you may want to interact with the metrics that define that service graph directly. You may want to, for example, programmatically analyze how your services are interconnected and build downstream applications that use this information.

Instant Queries

An instant query gives a single value at the end of the selected time range. Instant queries are quicker to execute and it often easier to understand their results. We prefer them in some scenarios:

Connectivity between services

Show the total calls in the last 7 days for every client/server pair:

sum(increase(traces_service_graph_request_server_seconds_count{}[7d])) by (server, client) > 0

Use this query to only see when a single service is the server:

sum(increase(traces_service_graph_request_server_seconds_count{server="foo"}[7d])) by (client) > 0

If you’d like to only see when a single service is the client:

sum(increase(traces_service_graph_request_server_seconds_count{client="foo"}[7d])) by (server) > 0

In all of the above queries, you can adjust the interval to change the amount of time this is calculated for. So if you wanted the same analysis done over one day:

sum(increase(traces_service_graph_request_server_seconds_count{}[1d])) by (server, client) > 0

Range queries

Range queries are nice for calculating service graph info over a time range instead of a single point in time.

Rates over time between services

Taking two of the queries above, we can request the rate over time that any given service acted as the client or server:

sum(rate(traces_service_graph_request_server_seconds_count{server="foo"}[5m])) by (client) > 0

sum(rate(traces_service_graph_request_server_seconds_count{client="foo"}[5m])) by (server) > 0

Notice that our interval dropped to 5m. This is so we only calculate the rate over the past 5 minutes which creates a more responsive graph.

Latency percentiles over time between services

These queries provide latency quantiles for the above rate. If you’re interested in how the latency changed over time between any two services we could use these. In the following query the .9 means we’re calculating the 90th percentile. Adjust this value if you want to calculate a different percentile for latency, for example, p50, p95, p99.

histogram_quantile(.9, sum(rate(traces_service_graph_request_server_seconds_bucket{client="foo"}[5m])) by (server, le))

Using the optional metric for latency of a messaging system to see potential middleware latencies:

histogram_quantile(.9, sum(rate(traces_service_graph_request_messaging_system_seconds_bucket{}[5m])) by (client, server, le))