Documentation Grafana Cloud Platform Security and account management Account management Grafana Cloud Stack Manage database configuration Configure Tempo trace metrics

Configure Tempo trace metrics

The Tempo metrics generator produces span metrics and service graph metrics from your traces. Application Observability uses these metrics to display RED (rate, errors, duration) metrics for your services. You can configure the metrics generator using the Grafana Cloud user interface (UI), from the Database configuration app.

Valid configuration changes will start applying as soon as you save. A status badge shows whether your latest change has finished rolling out. Invalid configurations will show errors and you will not be allowed to save an invalid configuration.

Before you begin

You need the Admin role on the stack to access the database configurations app.

Access the UI

To access the Traces configuration UI:

1. In your Grafana Cloud stack, in the main menu, expand Administration.
2. Select Database configurations.
3. Select Traces.

Metrics-generation is disabled by default. The Traces configuration page displays the Activate button.

If metrics generation has been activated, the configuration page opens on the Details tab. Four other tabs let you edit the configuration: Dimensions, Histogram buckets, Filter rules, and Advanced.

Activate metrics generation

If no configuration exists yet, the only available action is to activate metrics generation.

1. Click Activate.
2. Review the usage notice: Grafana Cloud uses the metrics generated from your traces, and they count toward your current usage and bill for this stack. There’s no additional cost for Grafana Cloud Free accounts.
3. If you previously had custom dimensions applied as labels through Grafana Support, review the warning that activating metrics generation overwrites that configuration.
4. Select the checkbox to acknowledge that activating automatic metrics generation incurs additional usage toward your bill.
5. Click Activate to confirm.

Activating creates the initial configuration with a default filter rule that scopes span metrics and service graphs to spans that enter a service (refer to Filter rules).

Configure dimensions

Dimensions are span attributes added to your span metrics and service graph metrics (and to traces_target_info), so you can filter and group by them. Each dimension increases series cardinality.

A fixed set of essential dimensions,service, span_name, span_kind, status_code, and status_message, are always included and aren’t shown in this list.

To add a dimension:

1. Click the Dimensions tab.

2. In the Add an attribute field, search for an attribute name or type a custom one, then select or add it. When Tempo can suggest attribute names, they appear as you type; otherwise, type the attribute name directly.

   Note

   Tempo returns only a limited number of suggested attribute names. If the list looks incomplete, narrow the results with a scope condition, or type the attribute name directly.

3. For each dimension, use the Span metrics and Service graph switch to choose which metrics include it.

4. You can add additional attributes as dimensions.

5. You can use the Search field to filter the list, or click the trash icon on a row to remove a dimension.

6. To discard your edits and revert to the last saved configuration, click Reset changes.

7. Click Save configuration. A confirmation dialog box opens:

   • It warns that saving applies your changes to live production systems.
   • If Application Observability is using this configuration, it warns that your changes may affect its dashboards and features.
   • It reminds you that Terraform may also manage this configuration, so confirm your change doesn’t conflict with your team’s configuration policies.
   • If someone else changed the configuration since you loaded the page, it warns that saving would overwrite that change.

8. Click Save.

Any client-side validation issues are shown as advisory warnings below the form; they may still be rejected when you save, since the server makes the final decision.

If you navigate away from the page with unsaved changes, a dialog box asks you to confirm before discarding them.

Configure Histogram buckets

Configure how span-duration histograms are bucketed for span metrics and service graphs.

Histogram type

1. Click the Histogram buckets tab.

2. Click the Histogram type card for the type you want:

   • Classic histograms - fixed bucket boundaries, configured with presets or custom values. Best for predictable latency patterns.
   • Native histograms - exponentially spaced buckets that automatically adapt to your data distribution, with more efficient storage and better percentile accuracy.
   • Both - generates classic and native histograms simultaneously. Useful when migrating between types, but doubles the number of buckets and increases costs. A confirmation dialog box opens when you select this option.

3. To configure classic histograms:

   1. Under Apply to, choose whether the bucket boundaries apply to Span metrics, Service graph, or Both. If the two currently have different boundaries, Both is disabled until you resolve the difference (pick one existing set as a starting template).

   2. Under Bucket granularity, select a preset:

      • Coarse (6 buckets; day-to-day dashboards, high-traffic APIs, cost control).
      • Balanced (10 buckets; enough detail for 95th/99th latency SLOs).
      • Detailed (14 buckets; deep performance work, tail-latency hunts).
      • select Advanced to define custom bucket boundaries.

   3. If you selected **Advanced"

      1. Click New bucket.

      2. In the Create new histogram buckets dialog box, you can add, edit, or remove individual bucket boundaries (in seconds). Guidance for common latency tiers is shown alongside the editor, for example:

         • Low latency services: smaller buckets, such as 0.001, 0.005, 0.01, 0.025, 0.05, 0.1
         • Medium latency services: mid-range buckets, such as 0.1, 0.25, 0.5, 1, 2.5, 5
         • High latency services: larger buckets, such as 5, 10, 30, 60, 120
         • Database operations: for example, 0.001, 0.01, 0.1, 1, 10
         • External API calls: for example, 0.1, 0.5, 1, 5, 10, 30

      Switching from custom (Advanced) boundaries to a preset discards your custom values - a confirmation dialog box opens first.

4. To configure native histograms:

   1. Set the Max bucket number - the maximum number of buckets to create (default: 16, maximum: 100,000).

5. Click Save configuration. A confirmation dialog box opens:

   • It warns that saving applies your changes to live production systems.
   • If Application Observability is using this configuration, it warns that your changes may affect its dashboards and features.
   • It reminds you that Terraform may also manage this configuration, so confirm your change doesn’t conflict with your team’s configuration policies.
   • If someone else changed the configuration since you loaded the page, it warns that saving would overwrite that change.

6. Click Save.

Configure filter rules

Filter rules let you be selective about which spans generate metrics. Every rule can apply to span metrics, service graphs, or both.

New configurations start with a default rule that includes only spans with a span kind of SERVER, CONSUMER, CLIENT, or PRODUCER, that is, spans that represent an entry point into a service. If this rule isn’t present, the page shows a notice with an Add default rules button so you can restore it. Without it, metrics are generated for every span.

Create a custom filter rule

1. Click the Filter rules tab.

2. Click Create filter rule.

3. In the Create filter rule dialog box, from the Match menu, select how the rule combines its attributes:

   • Include - keep only spans matching all of these attributes.
   • Include any - keep spans matching any of these attributes.
   • Exclude - drop spans matching these attributes.

1. Under Type, select either Strict (exact match) or Regex (regular-expression match).

1. Add one or more Attributes (key and value) to match on.

2. Under Apply to, select Span metrics, Service graph, or both switches.

3. Click Create. You can configure up to 1000 filter-policy attributes in total, of which up to 100 can use regular expression matching.

4. Click Save configuration. A confirmation dialog box opens:

   • It warns that saving applies your changes to live production systems.
   • If Application Observability is using this configuration, it warns that your changes may affect its dashboards and features.
   • It reminds you that Terraform may also manage this configuration, so confirm your change doesn’t conflict with your team’s configuration policies.
   • If someone else changed the configuration since you loaded the page, it warns that saving would overwrite that change.

5. Click Save.

Add rules from the catalog

1. Click Add from catalog to open a list of recommended noise-reduction rules, for example excluding health-check endpoints, metrics-scrape traffic, spans explicitly marked to skip, static assets, favicon requests, and robots.txt requests.
2. Click Add next to a rule to add it, or Remove if it’s already applied. The catalog dialog box stays open so you can add several rules in one pass.

Modify rules

1. You can edit or remove a rule from the filter rules table, and toggle each rule’s Span metrics / Service graph columns directly.

2. Click Save configuration. A confirmation dialog box opens:

   • It warns that saving applies your changes to live production systems.
   • If Application Observability is using this configuration, it warns that your changes may affect its dashboards and features.
   • It reminds you that Terraform may also manage this configuration, so confirm your change doesn’t conflict with your team’s configuration policies.
   • If someone else changed the configuration since you loaded the page, it warns that saving would overwrite that change.

3. Click Save.

Configure processors (Advanced tab)

If you’ve configured dimensions or filter rules for span metrics or service graphs, but haven’t enabled the matching processor, a warning banner reminds you that they won’t take effect until you do.

1. Click the Advanced tab.

2. Under Processors, select which metrics-generator processors are enabled for this Tempo instance:

   • span-metrics - RED metrics (rate, errors, duration) per span.
   • span-metrics-latency - latency histograms for span metrics.
   • span-metrics-count - request/span counts for span metrics.
   • span-metrics-size - span size metrics.
   • service-graphs - service-to-service edges (rate, errors, duration between services).
   • service-graphs-request - Request and failed-request counters for service graphs.
   • service-graphs-latency - Client and server latency histograms for service graphs.
   • service-graphs-connection-info - Connection-info gauge per service edge. Off by default, adds to service-graphs.
   • host-info - a per-host info metric keyed by configured host identifiers.

   To learn more about span metrics subprocessors, refer to the metrics-generator documentation.
   To learn more about service graphs subprocessors, refer to the service graph documentation.

3. Click Save configuration. A confirmation dialog box opens:

   • It warns that saving applies your changes to live production systems.
   • If Application Observability is using this configuration, it warns that your changes may affect its dashboards and features.
   • It reminds you that Terraform may also manage this configuration, so confirm your change doesn’t conflict with your team’s configuration policies.
   • If someone else changed the configuration since you loaded the page, it warns that saving would overwrite that change.

4. Click Save.

View current configuration

The Details tab is read-only and shows the status of your saved configuration:

• Saved Configuration
  • Saved configuration details: when it was created, its current generation number, when it was last edited, and what it was last edited with (for example, the Database configuration app, or another tool).
• Reconciliation
  • A status badge - Reconciled (your latest change is live), Reconciling new configuration (still rolling out), Failed, Not yet reconciled, or No configuration saved yet.
  • If the status is Failed, expand the failed entry to read its reason. Update your configuration to address that reason, then click Save configuration again - this submits a new generation for reconciliation. Resubmitting the same configuration unchanged is unlikely to resolve a failure. If the reason isn’t clear, or the failure persists after you’ve addressed it, open a support ticket.
• Environment
  • Whether Application Observability is currently using this configuration (In use or Not in use).
  • A collapsible Resource details section with lower-level identifiers (UID, resource version, namespace, labels).

Delete a configuration

1. To remove your Tempo metrics-generator configuration entirely, click the trash icon in the page header. A confirmation dialog box opens:

   • It warns that this permanently deletes the configuration for this stack.
   • If Application Observability is using this configuration, it warns that deleting it breaks its span metrics and service graphs.
   • It notes that the stack falls back to the Tempo default metrics-generator behavior, and that Terraform may re-create the configuration if it manages it.

2. Type delete to confirm.

3. Click Delete.

Deactivate and reactivate metrics generation

You can temporarily turn metrics generation off without losing your configuration:

1. Click Deactivate.
2. Review the notice that deactivating stops producing span metrics and service graph metrics from your traces. Existing metrics data isn’t deleted.
3. Select the checkbox to acknowledge that this stops generating metrics from traces.
4. Click Deactivate to confirm.

While metrics generation is deactivated, your existing configuration is shown read-only on every editing tab, and the only available action is Reactivate.

To reactivate:

1. Click Reactivate.
2. Review the usage notice (the same billing acknowledgment shown during activation).
3. Select the checkbox to acknowledge the additional usage.
4. Click Reactivate to confirm.

Reactivating resumes producing metrics using your existing configuration; it doesn’t reset your dimensions, histogram buckets, filter rules, or advanced settings.