Grafana Cloud Asserts prerequisites and troubleshooting
This topic describes the prerequisites you must meet to use Asserts. If you meet the prerequisites and Asserts isn’t working, refer to the troubleshooting section of this topic.
Infrastructure metrics
Asserts scans your metrics to identify infrastructure and USE metrics for the entities it discovers.
At the moment, Asserts can only collect infrastructure metrics from Kubernetes. To learn how to set up monitoring for your Kubernetes cluster, refer to Configure Kubernetes Monitoring.
If you’ve set up Kubernetes monitoring but aren’t seeing entities and information in Asserts, refer to Kubernetes Metrics for assistance with common issues.
Grafana Labs is working to expand support for other metric sources and custom metrics. If you’re interested in integrating your custom telemetry with Asserts, submit a support ticket.
RED metrics
Asserts attempts to match the RED metrics with the entities it detects through Kubernetes monitoring. By default, Asserts supports Grafana Cloud Application Observability and Istio.
To collect RED metrics, you need to instrument your applications with OpenTelemetry and send the data to Grafana Cloud.
If you can’t see your OpenTelemetry instrumented services in Asserts, refer to RED metrics for assistance with common issues.
Troubleshooting
This section provides troubleshooting tips for Kubernetes metrics and RED metrics.
Kubernetes metrics
If Asserts isn’t discovering entities or if you’re experiencing empty panels in your dashboards, it may be because Grafana Cloud Adaptive Metrics is dropping or aggregating metrics or labels that Asserts needs. If Adaptive Metrics is affecting the required metrics, you need to remove them from Adaptive Metrics. To learn how to remove metrics from Adaptive Metrics, refer to Recommendation exemptions.
The table below shows the metrics and labels necessary for Asserts to monitor your Kubernetes environment. If the labels are present but issues persist, open a support ticket for further assistance.
Metric name | Required labels | ||
---|---|---|---|
kube_pod_info | cluster, namespace, node, pod | ||
kube_pod_owner | cluster, namespace, node, owner_kind, owner_name | ||
kube_pod_container_resource_requests | cluster, namespace, pod, container, resource | ||
kube_pod_status_phase | cluster, namespace, pod, phase | ||
kube_replicaset_owner | cluster, namespace, replicaset, owner_name, owner_kind | ||
kube_pod_container_info | cluster, namespace, container, image_id | ||
kube_pod_container_resource_limits | cluster, namespace, pod, container, resource | ||
kube_configmap_metadata_resource_version | cluster, namespace, configmap | ||
kube_secret_metadata_resource_version | cluster, namespace, secret | ||
kube_deployment_metadata_generation | cluster, namespace, deployment | statefulset | daemonset |
kube_node_name | cluster, node, instance | ||
kube_node_info | cluster, provider_id, node | ||
kubelet_node_name | cluster, node, instance | ||
AWS | |||
kube_node_labels | label_beta_kubernetes_io_instance_type, and label_eks_amazonaws_com_nodegroup or | ||
label_karpenter_sh_nodepool or | |||
label_alpha_eksctl_io_cluster_name, label_alpha_eksctl_io_nodegroup_name or | |||
label_ec2_amazonaws_com_Name, label_ec2_amazonaws_com_aws_autoscaling_groupName or | |||
label_ec2_amazonaws_com_name, label_ec2_amazonaws_com_aws_autoscaling_group_name or | |||
label_k8s_io_cloud_provider_aws | |||
GCP | |||
kube_node_labels | label_node_kubernetes_io_instance_type, label_cluster_name, label_cloud_google_com_gke_nodepool | ||
Azure | |||
kube_node_labels | label_agentpool, label_kubernetes_azure_com_cluster | ||
kube_node_status_allocatable | cluster, node, resource |
Container resource utilization observability
The following table lists metrics and labels required for Kubernetes container resource utilization observability.
Metric name | Required labels |
---|---|
container_cpu_cfs_throttled_periods_total | cluster, namespace, pod, container, node |
container_cpu_cfs_periods_total | cluster, namespace, pod, container, node |
container_memory_working_set_bytes | cluster, namespace, pod, container, node |
container_memory_usage_bytes | cluster, namespace, pod, container, node |
container_memory_cache | cluster, namespace, pod, container, node |
RED metrics troubleshooting
For Asserts to associate the RED metrics with the Kubernetes entities it identifies, the entities must have labels that specify their source. For instance, span metrics require labels such as k8s.namespace.name
, k8s.cluster.name
, and k8s.pod.name
.
You can use the Kubernetes Attributes Process to assign these labels. Make sure you follow the Kubernetes monitoring recommendations.
If you still encounter problems, submit a support ticket for further assistance.