Configure scraping of application Pod metrics
Kubernetes Monitoring does not scrape application Prometheus metrics by default, but you can configure Grafana Agent to also scrape application Prometheus metrics, like those available at the standard /metrics
endpoint on pods.
To add a scrape job targeting all /metrics
endpoints on your cluster pods, do the following:
Add the following to the bottom of your Agent scrape config:
. . . - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: # Example relabel to scrape only Pods that have # "example.io/should_be_scraped = true" annotation. # - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped] # action: keep # regex: true # # Example relabel to customize metric path based on Pod # "example.io/metric_path = <metric path>" annotation. # - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path] # action: replace # target_label: __metrics_path__ # regex: (.+) # # Example relabel to scrape only single, desired port for the Pod # based on Pod "example.io/scrape_port = <port>" annotation. # - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port] # action: replace # regex: ([^:]+)(?::\d+)?;(\d+) # replacement: $1:$2 # target_label: __address__ # Expose Pod labels as metric labels - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # Expose Pod namespace as metric namespace label - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace # Expose Pod name as metric name label - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod
This config adds every defined Pod container port to Agent’s scrape targets, discovered using Agent’s Kubernetes service discovery mechanism. You can optionally uncomment the relevant sections to customize the metrics path (the default is
/metrics
), specify a sample port, or use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests. To learn more, see the examples in the official Prometheus project repo.To learn more about configuring Agent, see Configure Grafana Agent from the Agent docs. To learn more about available
kubernetes_sd_configs
labels and roles (we used thepod
role here), see kubernetes_sd_config from the Prometheus docs.Deploy the updated config into your cluster using
kubectl apply -f
:kubectl apply -f <YOUR-CONFIGMAP>.yaml
Restart the agent to pick up the config changes.
kubectl rollout restart deployment/grafana-agent
For a complete example, see Monitor an app on Kubernetes using Grafana Agent.