Grafana Cloud Grafana Kubernetes Monitoring Other configuration methods Scrape application metrics

Configure scraping of application Pod metrics

Kubernetes Monitoring does not scrape application Prometheus metrics by default, but you can configure Grafana Agent to also scrape application Prometheus metrics, like those available at the standard /metrics endpoint on pods.

To add a scrape job targeting all /metrics endpoints on your cluster pods, do the following:

1. Add the following to the bottom of your Agent scrape config:

   . . .
   - job_name: "kubernetes-pods"
       kubernetes_sd_configs:
         - role: pod
       relabel_configs:
         # Example relabel to scrape only Pods that have
         # "example.io/should_be_scraped = true" annotation.
         #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
         #    action: keep
         #    regex: true
         #
         # Example relabel to customize metric path based on Pod
         # "example.io/metric_path = <metric path>" annotation.
         #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
         #    action: replace
         #    target_label: __metrics_path__
         #    regex: (.+)
         #
         # Example relabel to scrape only single, desired port for the Pod
         # based on Pod "example.io/scrape_port = <port>" annotation.
         #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
         #    action: replace
         #    regex: ([^:]+)(?::\d+)?;(\d+)
         #    replacement: $1:$2
         #    target_label: __address__
         # Expose Pod labels as metric labels
         - action: labelmap
           regex: __meta_kubernetes_pod_label_(.+)
         # Expose Pod namespace as metric namespace label
         - source_labels: [__meta_kubernetes_namespace]
           action: replace
           target_label: namespace
         # Expose Pod name as metric name label
         - source_labels: [__meta_kubernetes_pod_name]
           action: replace
           target_label: pod
   

   This config adds every defined Pod container port to Agent’s scrape targets, discovered using Agent’s Kubernetes service discovery mechanism. You can optionally uncomment the relevant sections to customize the metrics path (the default is /metrics), specify a sample port, or use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests. To learn more, see the examples in the official Prometheus project repo.

   To learn more about configuring Agent, see Configure Grafana Agent from the Agent docs. To learn more about available kubernetes_sd_configs labels and roles (we used the pod role here), see kubernetes_sd_config from the Prometheus docs.

2. Deploy the updated config into your cluster using kubectl apply -f:

   kubectl apply -f <YOUR-CONFIGMAP>.yaml
   

3. Restart the agent to pick up the config changes.

   kubectl rollout restart deployment/grafana-agent
   

For a complete example, see Monitor an app on Kubernetes using Grafana Agent.