--- title: "Instrumentation quality | Grafana Cloud documentation" description: "A report on the quality of your instrumentation for the knowledge graph." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Instrumentation quality The instrumentation quality feature helps you identify and fix telemetry configuration issues that prevent you from getting maximum value from the knowledge graph. It analyzes your services and provides actionable recommendations to improve instrumentation completeness. You can view instrumentation quality in two ways: - **List view** - See all services with quality issues in the entity catalog’s Instrumentation quality tab - **Entity-level report** - View detailed quality information for a specific service ## View instrumentation quality for all services The Instrumentation quality tab in the entity catalog shows a complete overview of instrumentation health across all your services. To access the list view: 1. Navigate to **Observability > Entity catalog**. 2. Click the **Instrumentation quality** tab. The tab displays a badge showing the total number of services with quality issues. ### Understand quality scores Each service displays a quality score that represents instrumentation completeness: Expand table | Quality label | Score range | Description | |---------------|-------------|-------------------------------------------------------| | Perfect | 100% | Service has complete instrumentation | | Very good | 51-99% | Service has solid instrumentation with minor gaps | | Good | 26-50% | Service has acceptable instrumentation with some gaps | | Poor | 0-25% | Service has significant instrumentation gaps | Higher scores indicate better instrumentation quality. Services with lower scores are missing critical telemetry that limits knowledge graph capabilities like root cause analysis, entity relationships, and dashboard completeness. ### Filter by failing checks Use the **Failing checks** filter in the left sidebar to focus on specific instrumentation issues: - **All** - Show all services with any quality issues - **Cardinality breach** - Services sending high-cardinality span names - **K8s labels** - Services missing `k8s.cluster.name`, `k8s.namespace.name`, or `k8s.pod.name` resource attributes - **Mixed histograms** - Services mixing classic and native histogram types - **Service graph metrics k8s labels presence** - Service graph metrics missing the Kubernetes namespace label - **Service graph metrics presence** - Services with no service graph metrics - **Service logs** - Services with no logs - **Service name** - Services using slashes in `service.name` or `service.namespace` - **Service Profiles** - Services with no profiles - **Span metrics k8s labels presence** - Span metrics missing the Kubernetes namespace label - **Span metrics presence** - Services with no request (span) metrics - **Unique service source** - Services producing duplicate metrics from multiple sources Select a specific check to see only services affected by that issue. ### View failing checks The **Failing checks** column lists specific issues for each service. Each item shows: - The check name - A count indicating how many related issues exist Click **Quality report** for any service to see detailed recommendations and resolution steps. ### Export quality data Click **Download CSV** to export instrumentation quality data for services on the current page. The CSV includes: - Service name - Environment and namespace - Quality score and label - Complete list of failing checks Use the exported data for reporting, tracking quality improvements over time, or analyzing instrumentation issues across your organization. ## View instrumentation quality for a specific service The entity-level quality report provides detailed information about a single service’s instrumentation health. To access the quality report for a service: 1. Navigate to **Observability > Entity catalog**. 2. Click the **Instrumentation quality** tab or **Performance** tab. 3. Click a service name to open the entity drawer. 4. Click the **Quality report** tab. The tab displays a badge showing the number of failing checks for this service. ### Understand the quality report The quality report shows: - **Overall score** - Percentage score with an instrumentation rating (Perfect, Good, OK, Bad, or Incomplete instrumentation) - **Description** - Explanation of what the score means for this service - **Recommendation cards** - Specific issues with severity indicators and resolution steps ### Investigate failing checks Each recommendation card shows: - **Issue title** - Clear description of the problem - **Severity** - Critical or Recommended - **Explanation** - Why this issue matters - **Resolution steps** - Specific actions to fix the issue - **Documentation** - Link to detailed configuration instructions - **Explore query** - Link to query the issue directly (when applicable) Click **Explore query** to investigate the issue using Grafana Explore. This runs a query showing the specific metrics or labels that are missing or misconfigured. ## Common quality issues and resolutions The following sections describe common instrumentation quality issues and how to resolve them. ### Add span metrics Span metrics power Service Overview dashboards and provide request rate, error rate, and latency information. Without span metrics, Service Overview dashboards appear empty and you can’t monitor service performance effectively. To address this issue: - Enable [Grafana Cloud metrics generation](/docs/grafana-cloud/send-data/traces/configure/metrics-generator/) to automatically generate span metrics from your traces - Alternatively, configure your OpenTelemetry Collector to generate span metrics using the [`spanmetrics` connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector) - Ensure span metrics are exported to your Grafana Cloud Prometheus endpoint - Verify metrics are named `traces_spanmetrics_calls_total`, `calls_total`, or `traces_span_metrics_calls_total` ### Add service graph metrics Service graph metrics define relationships between services and power the entity graph. Without service graph metrics, the knowledge graph can’t show how services communicate or trace request paths through your system. To address this issue: - Enable [Grafana Cloud metrics generation](/docs/grafana-cloud/send-data/traces/configure/metrics-generator/) to automatically generate service graph metrics from your traces - Alternatively, configure your OpenTelemetry Collector to generate service graph metrics using the [`servicegraph` connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector) - Ensure service graph metrics include required labels like `client` and `server` - For Kubernetes environments, include `client_k8s_cluster_name`, `server_k8s_cluster_name`, `client_k8s_namespace_name`, and `server_k8s_namespace_name` labels ### Add missing Kubernetes labels Kubernetes labels like `k8s_namespace_name`, `k8s_cluster_name`, and `k8s_pod_name` enable the knowledge graph to link application services with infrastructure components. Without these labels, services and their underlying Pods appear as separate entities without relationships. To address this issue: - Use the recommended [Grafana Kubernetes Monitoring Helm chart](/docs/grafana-cloud/monitor-applications/application-observability/collector/grafana-alloy-kubernetes/) - Alternatively, configure the [k8sattributeprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md) in your OpenTelemetry Collector - Ensure namespace labels are consistent across span metrics and service graph metrics ### Kubernetes attributes missing on span or service graph metrics This check can also report that span metrics or service graph metrics are missing the `k8s_namespace_name` attribute even when the service’s resource attributes (`target_info`) already include it. In that case the Kubernetes attribute is set on the resource but is not carried through to the generated metrics, so some service relationships and Application-to-Kubernetes navigation are incomplete. This is usually a metrics-generation configuration issue rather than a missing resource attribute. Ensure the Kubernetes attributes are applied before metrics are generated from spans, either by using the recommended Helm chart or by ordering the `k8sattributes` processor ahead of metrics generation in your Grafana Alloy or an OpenTelemetry Collector pipeline. ### Reduce high-cardinality metrics High-cardinality metrics significantly increase storage costs and can impact query performance. Cardinality breaches occur when metrics have too many unique label combinations, often caused by including dynamic values like user IDs, timestamps, or session tokens in metric labels. To address this issue: - Review your metric labels and remove high-cardinality values - Use generic patterns or placeholders instead of specific values in span names (for example, `/my-api/v1/` instead of `/my-api/v1/15a63467`) - Avoid including unique identifiers in metric labels - Configure the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) to normalize high-cardinality labels - Refer to [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) for best practices ### Don’t mix histogram types The knowledge graph supports native and classic histograms, but each service must use only one type. If a service uses both types, only data from the native histogram appears, leading to incomplete metrics and dashboards. To address this issue: - Choose either native or classic histograms for your service, with native being the preferred - Configure your OpenTelemetry Collector to use a consistent histogram type - If using the `spanmetrics` connector, set the histogram configuration explicitly - Verify all instrumentation libraries for the service use the same histogram type ### Use a unique service source A service produces duplicate metrics when the same metric is generated from more than one source. This most commonly happens when an application is instrumented with both tracing and eBPF. Duplicate metrics distort rates and counts and make service behavior difficult to analyze. To address this issue: - Determine whether the service produces the same metric from multiple pipelines, such as both traces and eBPF - To stop Tempo from generating span metrics from traces, add the resource attribute `span.metrics.skip=true` to your trace data - Remove or disable any other redundant collector or instrumentation that duplicates the same telemetry ### Add the missing service.namespace attribute The knowledge graph uses the `service.namespace` attribute to filter and group by a service’s namespace. To address this issue, edit these attributes in your [instrumentation setup](/docs/grafana-cloud/send-data/otlp/send-data-otlp/#opentelemetry-environment-variables). ### Avoid slashes in service name and namespace attributes Avoid using slashes (/) in the `service.name` or `service.namespace` attributes. People often use slashes to namespace services. The OpenTelemetry specification has a specific namespace attribute. If you use the namespace attribute correctly, you can benefit from advanced group and filter features in the knowledge graph. To address this issue, edit these attributes in your [instrumentation setup](/docs/grafana-cloud/send-data/otlp/send-data-otlp/#opentelemetry-environment-variables). ### Send missing logs for your service Your service is missing logs. Send logs to help you debug and monitor your service. To capture and send your application logs to Grafana Cloud, use one of the following methods: - Use the recommended instrumentation set up for the knowledge graph - Send logs to [Loki](/docs/grafana-cloud/send-data/logs/) or the [OTLP endpoint](/docs/grafana-cloud/send-data/otlp/send-data-otlp/) ### Send missing profiles for your service Your service is missing profiles. Send profiling information to help you debug and monitor your service. To capture and send your application profile data to Grafana Cloud, [instrument your application for profiling data](/docs/grafana-cloud/monitor-applications/profiles/send-profile-data/). ## Next steps - [Prerequisites and signal requirements](/docs/grafana-cloud/knowledge-graph/get-started/prerequisites/) - Review what metrics and labels the knowledge graph requires - [Troubleshoot setup issues](/docs/grafana-cloud/knowledge-graph/get-started/troubleshooting/) - Resolve issues with entity discovery and RED metrics - [Configure Application Observability metrics](/docs/grafana-cloud/knowledge-graph/get-started/manage-datasets/application/application-metrics/) - Manage metrics generation and deduplication settings