Grafana Cloud Monitor infrastructure Kubernetes Monitoring Configure Kubernetes Monitoring Configure manually for applications Scrape application Pod metrics

Scrape application Pod metrics

Kubernetes Monitoring does not scrape application Prometheus metrics by default. However, you can configure Grafana Agent to scrape application Prometheus metrics, such as those available at the standard /metrics endpoint on Pods.

To add a scrape job targeting all /metrics endpoints on your cluster Pods, complete the following steps:

1. Add the following to the end of your Agent scrape config:

   bash

   . . .
   - job_name: "kubernetes-pods"
       kubernetes_sd_configs:
         - role: pod
       relabel_configs:
         # Example relabel to scrape only Pods that have
         # "example.io/should_be_scraped = true" annotation.
         #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
         #    action: keep
         #    regex: true
         #
         # Example relabel to customize metric path based on Pod
         # "example.io/metric_path = <metric path>" annotation.
         #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
         #    action: replace
         #    target_label: __metrics_path__
         #    regex: (.+)
         #
         # Example relabel to scrape only single, desired port for the Pod
         # based on Pod "example.io/scrape_port = <port>" annotation.
         #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
         #    action: replace
         #    regex: ([^:]+)(?::\d+)?;(\d+)
         #    replacement: $1:$2
         #    target_label: __address__
         # Expose Pod labels as metric labels
         - action: labelmap
           regex: __meta_kubernetes_pod_label_(.+)
         # Expose Pod namespace as metric namespace label
         - source_labels: [__meta_kubernetes_namespace]
           action: replace
           target_label: namespace
         # Expose Pod name as metric name label
         - source_labels: [__meta_kubernetes_pod_name]
           action: replace
           target_label: pod

   This config adds every defined Pod container port to the Agent’s scrape targets, which are discovered using Agent’s Kubernetes service discovery mechanism. You can optionally do any of the following:

   • Uncomment the relevant sections to customize the metrics path (the default is /metrics).
   • Specify a sample port.
   • Use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests.

   To learn more, refer to the Prometheus examples.

   For more about configuring Agent, refer to Configure Grafana Agent static mode. For more about available kubernetes_sd_configs labels and roles (we used the pod role here), refer to kubernetes_sd_config.

2. Deploy the updated config into your Cluster using kubectl apply -f:

   bash

   kubectl apply -f <YOUR-CONFIGMAP>.yaml

3. Restart the agent to pick up the config changes.

   bash

   kubectl rollout restart deployment/grafana-agent

For a complete example, refer to Monitor an app on Kubernetes using Grafana Agent.