--- title: "Troubleshoot data issues | Grafana Cloud documentation" description: "Troubleshoot data issues with the knowledge graph" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Troubleshoot data issues This topic provides guidance for troubleshooting data issues in the knowledge graph. ## Required metrics and labels If the knowledge graph isn’t discovering entities or if you’re experiencing empty panels in your dashboards, it may be because Grafana Cloud Adaptive Metrics is dropping or aggregating metrics or labels that the knowledge graph needs. If Adaptive Metrics is affecting the required metrics, you need to remove them from Adaptive Metrics. To learn how to remove metrics from Adaptive Metrics, refer to [Recommendation exemptions](/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/control-metrics-usage-via-adaptive-metrics/adaptive-metrics-api/#recommendation-exemptions). ### Application Observability required metrics and labels For an overview of the metrics and labels necessary for the knowledge graph to monitor your environment when using Application Observability, refer to [Application Observability required metrics and labels](/docs/grafana-cloud/monitor-applications/application-observability/setup/metrics-labels/). If the labels are present but issues persist, open a support ticket for further assistance. For more information on how to send `traces_host_info`, refer to [Host-hours pricing](/docs/grafana-cloud/monitor-applications/application-observability/pricing/). ### Kubernetes metrics The table below shows the metrics and labels necessary for the knowledge graph to monitor your Kubernetes environment. If the labels are present but issues persist, open a support ticket for further assistance. Expand table | Metric name | Required labels | | | |----------------------------------------------|-------------------------------------------------------------------------------------------------------------|-------------|-----------| | kube\_pod\_info | cluster, namespace, node, pod | | | | kube\_pod\_owner | cluster, namespace, node, owner\_kind, owner\_name | | | | kube\_pod\_container\_resource\_requests | cluster, namespace, pod, container, resource | | | | kube\_pod\_status\_phase | cluster, namespace, pod, phase | | | | kube\_replicaset\_owner | cluster, namespace, replicaset, owner\_name, owner\_kind | | | | kube\_pod\_container\_info | cluster, namespace, container, image\_id | | | | kube\_pod\_container\_resource\_limits | cluster, namespace, pod, container, resource | | | | kube\_configmap\_metadata\_resource\_version | cluster, namespace, configmap | | | | kube\_secret\_metadata\_resource\_version | cluster, namespace, secret | | | | kube\_deployment\_metadata\_generation | cluster, namespace, deployment | statefulset | daemonset | | kube\_node\_info | cluster, node | | | | kubelet\_node\_name | cluster, node, instance | | | | **AWS** | | | | | kube\_node\_labels | label\_beta\_kubernetes\_io\_instance\_type, and label\_eks\_amazonaws\_com\_nodegroup or | | | | | label\_karpenter\_sh\_nodepool or | | | | | label\_alpha\_eksctl\_io\_cluster\_name, label\_alpha\_eksctl\_io\_nodegroup\_name or | | | | | label\_ec2\_amazonaws\_com\_Name, label\_ec2\_amazonaws\_com\_aws\_autoscaling\_groupName or | | | | | label\_ec2\_amazonaws\_com\_name, label\_ec2\_amazonaws\_com\_aws\_autoscaling\_group\_name or | | | | | label\_k8s\_io\_cloud\_provider\_aws | | | | **GCP** | | | | | kube\_node\_labels | label\_node\_kubernetes\_io\_instance\_type, label\_cluster\_name, label\_cloud\_google\_com\_gke\_nodepool | | | | **Azure** | | | | | kube\_node\_labels | label\_agentpool, label\_kubernetes\_azure\_com\_cluster | | | | kube\_node\_status\_allocatable | cluster, node, resource | | | ### Container resource utilization observability The following table lists metrics and labels required for Kubernetes container resource utilization observability. Expand table | Metric name | Required labels | |------------------------------------------------|------------------------------------------| | container\_cpu\_cfs\_throttled\_periods\_total | cluster, namespace, pod, container, node | | container\_cpu\_cfs\_periods\_total | cluster, namespace, pod, container, node | | container\_memory\_working\_set\_bytes | cluster, namespace, pod, container, node | | container\_memory\_usage\_bytes | cluster, namespace, pod, container, node | | container\_memory\_cache | cluster, namespace, pod, container, node | ## RED metrics troubleshooting For the knowledge graph to associate the RED metrics with the Kubernetes entities it identifies, the entities must have labels that specify their source. For instance, span metrics require labels such as `k8s.namespace.name`, `k8s.cluster.name`, and `k8s.pod.name`. You can use the [Kubernetes Attributes Process](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md) to assign these labels. Make sure you follow the [Kubernetes monitoring recommendations](/docs/grafana-cloud/monitor-applications/application-observability/collector/grafana-alloy-kubernetes/). If you still encounter problems, submit a support ticket for further assistance. ## Prometheus troubleshooting In addition to using Grafana Cloud Application Observability or Grafana Cloud Kubernetes Monitoring, you might use Prometheus to scrape some metrics. However, there are some guidelines to consider for the knowledge graph to work correctly. If you use a single Prometheus job to scrape multiple entities, it can create the following issues: - The knowledge graph might not be able to detect all your entities. - RED metrics might not get associated to entities. - RED metrics might get aggregated across workloads that share the same job. To avoid issues, we recommend the following: - Make the entities easily identifiable. You can do this by applying one of the following methods: - Try not to use a single job to scrape multiple services and instead use a job per service. - Identify your entities by adding a `service` label to your metrics. - If you are using annotation-based [Kubernetes service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) in your Prometheus configuration, you can use the following relabeling rules: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml source_labels: [__meta_kubernetes_pod_name] regex: ^(.*?)([-][a-zA-Z0-9]{5,10}(-[a-zA-Z0-9]{5})?|-[0-9]+)?$ target_label: service replacement: $1 ``` ## AWS troubleshooting The following sections list the metrics and labels necessary for the knowledge graph to discover Amazon Web Services (AWS) entities and build relationships. ### AWS Cloud Provider Observability entity discovery The following table lists the metrics and labels necessary for the knowledge graph to discover Amazon Web Services (AWS) entities from AWS Cloud Provider Observability. All AWS entities require the `asserts_env` and `asserts_site` labels for scoping. Amazon Web Services (AWS) Cloud Provider Observability metrics carry the Amazon Resource Name in the `name` label. The knowledge graph extracts the entity name from the Amazon Resource Name using pattern matching. Expand table | Entity type | Metric name | Required labels | Name derivation | |-------------------------------|---------------------------|-------------------------------------------------|----------------------------------------------------------------------| | Amazon EC2 Instance | aws\_ec2\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name last segment | | Amazon RDS Instance | aws\_rds\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name via pattern `^.+:db:(.+)$` | | AWS Application Load Balancer | aws\_applicationelb\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name path | | AWS Network Load Balancer | aws\_networkelb\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name path | | AWS Lambda function | aws\_lambda\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name via pattern `^.+:(.+)$` | | Amazon Simple Storage Service | aws\_s3\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name via pattern `arn:aws:s3:::(.+)$` | | Amazon Simple Queue Service | aws\_sqs\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name via pattern `^.+:(.+)$` | | Amazon DynamoDB table | aws\_dynamodb\_info | asserts\_env, asserts\_site, name | Extracted from Amazon Resource Name path | | Amazon ECS Service | aws*ecs** | asserts\_env, asserts\_site, namespace, service | Direct from `service` label | | Amazon API Gateway | aws*apigateway** | asserts\_env, asserts\_site, namespace, service | Direct from `service` label | ### Amazon RDS relationships The following table lists the metrics and labels necessary for the knowledge graph to build Amazon RDS entities and relationships. These metrics are generated from [span metrics sources](/docs/grafana-cloud/monitor-applications/application-observability/manual/configure/#span-metrics-source) and help identify relationships to Amazon RDS instances by matching `*.rds.amazonaws.com` hostname patterns in the required labels. Expand table | Metric name | Required labels | |---------------------------------------------------------|-------------------------| | traces\_service\_graph\_request\_client\_seconds\_count | client\_server\_address | | traces\_service\_graph\_request\_client\_seconds\_count | server | | traces\_span\_metrics\_calls\_total | server\_address | | traces\_span\_metrics\_calls\_total | net\_peer\_name | | traces\_spanmetrics\_calls\_total | net\_peer\_name | ## Azure troubleshooting The following table lists the metrics and labels necessary for the knowledge graph to discover Azure entities from Cloud Provider Observability. All Azure entities require the `asserts_env` and `asserts_site` labels for scoping. Azure metrics carry the `resourceName` label natively. The knowledge graph uses this label directly without derivation. Expand table | Entity type | Metric name | Required labels | Name derivation | |-----------------------|-----------------------------------------------------------------------------------------|-------------------------------------------|--------------------------| | Azure Virtual Machine | azure\_microsoft\_compute\_virtualmachines\_vmavailabilitymetric\_average\_count | asserts\_env, asserts\_site, resourceName | Direct from resourceName | | Azure Flexible Server | azure\_microsoft\_dbforpostgresql\_flexibleservers\_active\_connections\_average\_count | asserts\_env, asserts\_site, resourceName | Direct from resourceName | | Azure Flexible Server | azure\_microsoft\_dbformysql\_flexibleservers\_active\_connections\_average\_count | asserts\_env, asserts\_site, resourceName | Direct from resourceName | | Azure Blob Storage | azure\_microsoft\_storage\_storageaccounts\_blobservices\_blobcount\_average\_count | asserts\_env, asserts\_site, resourceName | Direct from resourceName | ## Google Cloud troubleshooting The following table lists the metrics and labels necessary for the knowledge graph to discover GCP entities from Cloud Provider Observability. All GCP entities require the `asserts_env` and `asserts_site` labels for scoping. Stackdriver metrics carry the name natively in the `instance_name` or `database_id` label. For Cloud SQL, the knowledge graph extracts the instance name from `database_id` using pattern matching. Expand table | Entity type | Metric name | Required labels | Name derivation | |-------------------------|----------------------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------| | Compute Engine Instance | stackdriver\_gce\_instance\_compute\_googleapis\_com\_instance\_cpu\_utilization | asserts\_env, asserts\_site, instance\_name | Direct from instance\_name | | Cloud SQL Instance | stackdriver\_cloudsql\_database\_cloudsql\_googleapis\_com\_database\_up | asserts\_env, asserts\_site, database\_id | Extracted from database\_id via pattern `.+:(.+)$` |