Documentation Grafana Cloud Knowledge graph Explore features Entity catalog Assess instrumentation quality

Instrumentation quality

The instrumentation quality feature helps you identify and fix telemetry configuration issues that prevent you from getting maximum value from the knowledge graph. It analyzes your services and provides actionable recommendations to improve instrumentation completeness.

You can view instrumentation quality in two ways:

• List view - See all services with quality issues in the entity catalog’s Instrumentation quality tab
• Entity-level report - View detailed quality information for a specific service

View instrumentation quality for all services

The Instrumentation quality tab in the entity catalog shows a complete overview of instrumentation health across all your services.

To access the list view:

1. Navigate to Observability > Entity catalog.
2. Click the Instrumentation quality tab.

The tab displays a badge showing the total number of services with quality issues.

Understand quality scores

Each service displays a quality score that represents instrumentation completeness:

Higher scores indicate better instrumentation quality. Services with lower scores are missing critical telemetry that limits knowledge graph capabilities like root cause analysis, entity relationships, and dashboard completeness.

Filter by failing checks

Use the Failing checks filter in the left sidebar to focus on specific instrumentation issues:

• All - Show all services with any quality issues
• Cardinality breach - Services sending high-cardinality span names
• K8s labels - Services missing k8s.cluster.name, k8s.namespace.name, or k8s.pod.name resource attributes
• Mixed histograms - Services mixing classic and native histogram types
• Service graph metrics k8s labels presence - Service graph metrics missing the Kubernetes namespace label
• Service graph metrics presence - Services with no service graph metrics
• Service logs - Services with no logs
• Service name - Services using slashes in service.name or service.namespace
• Service Profiles - Services with no profiles
• Span metrics k8s labels presence - Span metrics missing the Kubernetes namespace label
• Span metrics presence - Services with no request (span) metrics
• Unique service source - Services producing duplicate metrics from multiple sources

Select a specific check to see only services affected by that issue.

View failing checks

The Failing checks column lists specific issues for each service. Each item shows:

• The check name
• A count indicating how many related issues exist

Click Quality report for any service to see detailed recommendations and resolution steps.

Export quality data

Click Download CSV to export instrumentation quality data for services on the current page. The CSV includes:

• Service name
• Environment and namespace
• Quality score and label
• Complete list of failing checks

Use the exported data for reporting, tracking quality improvements over time, or analyzing instrumentation issues across your organization.

View instrumentation quality for a specific service

The entity-level quality report provides detailed information about a single service’s instrumentation health.

To access the quality report for a service:

1. Navigate to Observability > Entity catalog.
2. Click the Instrumentation quality tab or Performance tab.
3. Click a service name to open the entity drawer.
4. Click the Quality report tab.

The tab displays a badge showing the number of failing checks for this service.

Understand the quality report

The quality report shows:

• Overall score - Percentage score with an instrumentation rating (Perfect, Good, OK, Bad, or Incomplete instrumentation)
• Description - Explanation of what the score means for this service
• Recommendation cards - Specific issues with severity indicators and resolution steps

Investigate failing checks

Each recommendation card shows:

• Issue title - Clear description of the problem
• Severity - Critical or Recommended
• Explanation - Why this issue matters
• Resolution steps - Specific actions to fix the issue
• Documentation - Link to detailed configuration instructions
• Explore query - Link to query the issue directly (when applicable)

Click Explore query to investigate the issue using Grafana Explore. This runs a query showing the specific metrics or labels that are missing or misconfigured.

Common quality issues and resolutions

The following sections describe common instrumentation quality issues and how to resolve them.

Add span metrics

Span metrics power Service Overview dashboards and provide request rate, error rate, and latency information. Without span metrics, Service Overview dashboards appear empty and you can’t monitor service performance effectively.

To address this issue:

• Enable Grafana Cloud metrics generation to automatically generate span metrics from your traces
• Alternatively, configure your OpenTelemetry Collector to generate span metrics using the spanmetrics connector
• Ensure span metrics are exported to your Grafana Cloud Prometheus endpoint
• Verify metrics are named traces_spanmetrics_calls_total, calls_total, or traces_span_metrics_calls_total

Add service graph metrics

Service graph metrics define relationships between services and power the entity graph. Without service graph metrics, the knowledge graph can’t show how services communicate or trace request paths through your system.

To address this issue:

• Enable Grafana Cloud metrics generation to automatically generate service graph metrics from your traces
• Alternatively, configure your OpenTelemetry Collector to generate service graph metrics using the servicegraph connector
• Ensure service graph metrics include required labels like client and server
• For Kubernetes environments, include client_k8s_cluster_name, server_k8s_cluster_name, client_k8s_namespace_name, and server_k8s_namespace_name labels

Add missing Kubernetes labels

Kubernetes labels like k8s_namespace_name, k8s_cluster_name, and k8s_pod_name enable the knowledge graph to link application services with infrastructure components. Without these labels, services and their underlying Pods appear as separate entities without relationships.

To address this issue:

• Use the recommended Grafana Kubernetes Monitoring Helm chart
• Alternatively, configure the k8sattributeprocessor in your OpenTelemetry Collector
• Ensure namespace labels are consistent across span metrics and service graph metrics

Kubernetes attributes missing on span or service graph metrics

This check can also report that span metrics or service graph metrics are missing the k8s_namespace_name attribute even when the service’s resource attributes (target_info) already include it. In that case the Kubernetes attribute is set on the resource but is not carried through to the generated metrics, so some service relationships and Application-to-Kubernetes navigation are incomplete.

This is usually a metrics-generation configuration issue rather than a missing resource attribute. Ensure the Kubernetes attributes are applied before metrics are generated from spans, either by using the recommended Helm chart or by ordering the k8sattributes processor ahead of metrics generation in your Grafana Alloy or an OpenTelemetry Collector pipeline.

Reduce high-cardinality metrics

High-cardinality metrics significantly increase storage costs and can impact query performance. Cardinality breaches occur when metrics have too many unique label combinations, often caused by including dynamic values like user IDs, timestamps, or session tokens in metric labels.

To address this issue:

• Review your metric labels and remove high-cardinality values
• Use generic patterns or placeholders instead of specific values in span names (for example, /my-api/v1/<id> instead of /my-api/v1/15a63467)
• Avoid including unique identifiers in metric labels
• Configure the transform processor to normalize high-cardinality labels
• Refer to OpenTelemetry semantic conventions for best practices

Don’t mix histogram types

The knowledge graph supports native and classic histograms, but each service must use only one type. If a service uses both types, only data from the native histogram appears, leading to incomplete metrics and dashboards.

To address this issue:

• Choose either native or classic histograms for your service, with native being the preferred
• Configure your OpenTelemetry Collector to use a consistent histogram type
• If using the spanmetrics connector, set the histogram configuration explicitly
• Verify all instrumentation libraries for the service use the same histogram type

Use a unique service source

A service produces duplicate metrics when the same metric is generated from more than one source. This most commonly happens when an application is instrumented with both tracing and eBPF. Duplicate metrics distort rates and counts and make service behavior difficult to analyze.

To address this issue:

• Determine whether the service produces the same metric from multiple pipelines, such as both traces and eBPF
• To stop Tempo from generating span metrics from traces, add the resource attribute span.metrics.skip=true to your trace data
• Remove or disable any other redundant collector or instrumentation that duplicates the same telemetry

Add the missing service.namespace attribute

The knowledge graph uses the service.namespace attribute to filter and group by a service’s namespace.

To address this issue, edit these attributes in your instrumentation setup.

Avoid slashes in service name and namespace attributes

Avoid using slashes (/) in the service.name or service.namespace attributes. People often use slashes to namespace services. The OpenTelemetry specification has a specific namespace attribute. If you use the namespace attribute correctly, you can benefit from advanced group and filter features in the knowledge graph.

To address this issue, edit these attributes in your instrumentation setup.

Send missing logs for your service

Your service is missing logs. Send logs to help you debug and monitor your service.

To capture and send your application logs to Grafana Cloud, use one of the following methods:

• Use the recommended instrumentation set up for the knowledge graph
• Send logs to Loki or the OTLP endpoint

Send missing profiles for your service

Your service is missing profiles. Send profiling information to help you debug and monitor your service.

To capture and send your application profile data to Grafana Cloud, instrument your application for profiling data.