Scrape application Pod metrics
Kubernetes Monitoring does not scrape application Prometheus metrics by default. However, you can configure Grafana Agent to scrape application Prometheus metrics, such as those available at the standard /metrics
endpoint on Pods.
To add a scrape job targeting all /metrics
endpoints on your cluster Pods, complete the following steps:
Add the following to the end of your Agent scrape config:
bash. . . - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: # Example relabel to scrape only Pods that have # "example.io/should_be_scraped = true" annotation. # - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped] # action: keep # regex: true # # Example relabel to customize metric path based on Pod # "example.io/metric_path = <metric path>" annotation. # - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path] # action: replace # target_label: __metrics_path__ # regex: (.+) # # Example relabel to scrape only single, desired port for the Pod # based on Pod "example.io/scrape_port = <port>" annotation. # - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port] # action: replace # regex: ([^:]+)(?::\d+)?;(\d+) # replacement: $1:$2 # target_label: __address__ # Expose Pod labels as metric labels - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # Expose Pod namespace as metric namespace label - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace # Expose Pod name as metric name label - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod
This config adds every defined Pod container port to the Agent’s scrape targets, which are discovered using Agent’s Kubernetes service discovery mechanism. You can optionally do any of the following:
- Uncomment the relevant sections to customize the metrics path (the default is
/metrics
). - Specify a sample port.
- Use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests.
To learn more, refer to the Prometheus examples.
For more about configuring Agent, refer to Configure Grafana Agent static mode. For more about available
kubernetes_sd_configs
labels and roles (we used thepod
role here), refer to kubernetes_sd_config.- Uncomment the relevant sections to customize the metrics path (the default is
Deploy the updated config into your Cluster using
kubectl apply -f
:bashkubectl apply -f <YOUR-CONFIGMAP>.yaml
Restart the agent to pick up the config changes.
bashkubectl rollout restart deployment/grafana-agent
For a complete example, refer to Monitor an app on Kubernetes using Grafana Agent.
Related resources from Grafana Labs


