Grafana Cloud Asserts prerequisites and troubleshooting

This topic describes the prerequisites you must meet to use Asserts. If you meet the prerequisites and Asserts isn’t working, refer to the troubleshooting section of this topic.

Infrastructure metrics

Asserts scans your metrics to identify infrastructure and USE metrics for the entities it discovers.

At the moment, Asserts can only collect infrastructure metrics from Kubernetes. To learn how to set up monitoring for your Kubernetes cluster, refer to Configure Kubernetes Monitoring.

If you’ve set up Kubernetes monitoring but aren’t seeing entities and information in Asserts, refer to Kubernetes Metrics for assistance with common issues.

Grafana Labs is working to expand support for other metric sources and custom metrics. If you’re interested in integrating your custom telemetry with Asserts, submit a support ticket.

RED metrics

Asserts attempts to match the RED metrics with the entities it detects through Kubernetes monitoring. By default, Asserts supports Grafana Cloud Application Observability and Istio.

To collect RED metrics, you need to instrument your applications with OpenTelemetry and send the data to Grafana Cloud.

If you can’t see your OpenTelemetry instrumented services in Asserts, refer to RED metrics for assistance with common issues.

Troubleshooting

This section provides troubleshooting tips for Kubernetes metrics and RED metrics.

Kubernetes metrics

If Asserts isn’t discovering entities or if you’re experiencing empty panels in your dashboards, it may be because Grafana Cloud Adaptive Metrics is dropping or aggregating metrics or labels that Asserts needs. If Adaptive Metrics is affecting the required metrics, you need to remove them from Adaptive Metrics. To learn how to remove metrics from Adaptive Metrics, refer to Recommendation exemptions.

The table below shows the metrics and labels necessary for Asserts to monitor your Kubernetes environment. If the labels are present but issues persist, open a support ticket for further assistance.

Metric name	Required labels
kube_pod_info	cluster, namespace, node, pod
kube_pod_owner	cluster, namespace, node, owner_kind, owner_name
kube_pod_container_resource_requests	cluster, namespace, pod, container, resource
kube_pod_status_phase	cluster, namespace, pod, phase
kube_replicaset_owner	cluster, namespace, replicaset, owner_name, owner_kind
kube_pod_container_info	cluster, namespace, container, image_id
kube_pod_container_resource_limits	cluster, namespace, pod, container, resource
kube_configmap_metadata_resource_version	cluster, namespace, configmap
kube_secret_metadata_resource_version	cluster, namespace, secret
kube_deployment_metadata_generation	cluster, namespace, deployment	statefulset	daemonset
kube_node_name	cluster, node, instance
kube_node_info	cluster, provider_id, node
kubelet_node_name	cluster, node, instance
AWS
kube_node_labels	label_beta_kubernetes_io_instance_type, and label_eks_amazonaws_com_nodegroup or
	label_karpenter_sh_nodepool or
	label_alpha_eksctl_io_cluster_name, label_alpha_eksctl_io_nodegroup_name or
	label_ec2_amazonaws_com_Name, label_ec2_amazonaws_com_aws_autoscaling_groupName or
	label_ec2_amazonaws_com_name, label_ec2_amazonaws_com_aws_autoscaling_group_name or
	label_k8s_io_cloud_provider_aws
GCP
kube_node_labels	label_node_kubernetes_io_instance_type, label_cluster_name, label_cloud_google_com_gke_nodepool
Azure
kube_node_labels	label_agentpool, label_kubernetes_azure_com_cluster
kube_node_status_allocatable	cluster, node, resource

Container resource utilization observability

The following table lists metrics and labels required for Kubernetes container resource utilization observability.

Metric name	Required labels
container_cpu_cfs_throttled_periods_total	cluster, namespace, pod, container, node
container_cpu_cfs_periods_total	cluster, namespace, pod, container, node
container_memory_working_set_bytes	cluster, namespace, pod, container, node
container_memory_usage_bytes	cluster, namespace, pod, container, node
container_memory_cache	cluster, namespace, pod, container, node

RED metrics troubleshooting

For Asserts to associate the RED metrics with the Kubernetes entities it identifies, the entities must have labels that specify their source. For instance, span metrics require labels such as k8s.namespace.name, k8s.cluster.name, and k8s.pod.name.

You can use the Kubernetes Attributes Process to assign these labels. Make sure you follow the Kubernetes monitoring recommendations.

If you still encounter problems, submit a support ticket for further assistance.