Menu
Grafana Cloud

Grafana Cloud Asserts prerequisites and troubleshooting

This topic describes the prerequisites you must meet to use Asserts. If you meet the prerequisites and Asserts isn’t working, refer to the troubleshooting section of this topic.

Infrastructure metrics

Asserts scans your metrics to identify infrastructure and USE metrics for the entities it discovers.

At the moment, Asserts can only collect infrastructure metrics from Kubernetes. To learn how to set up monitoring for your Kubernetes cluster, refer to Configure Kubernetes Monitoring.

If you’ve set up Kubernetes monitoring but aren’t seeing entities and information in Asserts, refer to Kubernetes Metrics for assistance with common issues.

Grafana Labs is working to expand support for other metric sources and custom metrics. If you’re interested in integrating your custom telemetry with Asserts, submit a support ticket.

RED metrics

Asserts attempts to match the RED metrics with the entities it detects through Kubernetes monitoring. By default, Asserts supports Grafana Cloud Application Observability and Istio.

To collect RED metrics, you need to instrument your applications with OpenTelemetry and send the data to Grafana Cloud.

If you can’t see your OpenTelemetry instrumented services in Asserts, refer to RED metrics for assistance with common issues.

Troubleshooting

This section provides troubleshooting tips for Kubernetes metrics and RED metrics.

Kubernetes metrics

If Asserts isn’t discovering entities or if you’re experiencing empty panels in your dashboards, it may be because Grafana Cloud Adaptive Metrics is dropping or aggregating metrics or labels that Asserts needs. If Adaptive Metrics is affecting the required metrics, you need to remove them from Adaptive Metrics. To learn how to remove metrics from Adaptive Metrics, refer to Recommendation exemptions.

The table below shows the metrics and labels necessary for Asserts to monitor your Kubernetes environment. If the labels are present but issues persist, open a support ticket for further assistance.

Metric nameRequired labels
kube_pod_infocluster, namespace, node, pod
kube_pod_ownercluster, namespace, node, owner_kind, owner_name
kube_pod_container_resource_requestscluster, namespace, pod, container, resource
kube_pod_status_phasecluster, namespace, pod, phase
kube_replicaset_ownercluster, namespace, replicaset, owner_name, owner_kind
kube_pod_container_infocluster, namespace, container, image_id
kube_pod_container_resource_limitscluster, namespace, pod, container, resource
kube_configmap_metadata_resource_versioncluster, namespace, configmap
kube_secret_metadata_resource_versioncluster, namespace, secret
kube_deployment_metadata_generationcluster, namespace, deploymentstatefulsetdaemonset
kube_node_namecluster, node, instance
kube_node_infocluster, provider_id, node
kubelet_node_namecluster, node, instance
AWS
kube_node_labelslabel_beta_kubernetes_io_instance_type, and label_eks_amazonaws_com_nodegroup or
label_karpenter_sh_nodepool or
label_alpha_eksctl_io_cluster_name, label_alpha_eksctl_io_nodegroup_name or
label_ec2_amazonaws_com_Name, label_ec2_amazonaws_com_aws_autoscaling_groupName or
label_ec2_amazonaws_com_name, label_ec2_amazonaws_com_aws_autoscaling_group_name or
label_k8s_io_cloud_provider_aws
GCP
kube_node_labelslabel_node_kubernetes_io_instance_type, label_cluster_name, label_cloud_google_com_gke_nodepool
Azure
kube_node_labelslabel_agentpool, label_kubernetes_azure_com_cluster
kube_node_status_allocatablecluster, node, resource

Container resource utilization observability

The following table lists metrics and labels required for Kubernetes container resource utilization observability.

Metric nameRequired labels
container_cpu_cfs_throttled_periods_totalcluster, namespace, pod, container, node
container_cpu_cfs_periods_totalcluster, namespace, pod, container, node
container_memory_working_set_bytescluster, namespace, pod, container, node
container_memory_usage_bytescluster, namespace, pod, container, node
container_memory_cachecluster, namespace, pod, container, node

RED metrics troubleshooting

For Asserts to associate the RED metrics with the Kubernetes entities it identifies, the entities must have labels that specify their source. For instance, span metrics require labels such as k8s.namespace.name, k8s.cluster.name, and k8s.pod.name.

You can use the Kubernetes Attributes Process to assign these labels. Make sure you follow the Kubernetes monitoring recommendations.

If you still encounter problems, submit a support ticket for further assistance.