Grafana Cloud

Scrape application Pod metrics

Kubernetes Monitoring does not scrape application Prometheus metrics by default. However, you can configure Grafana Agent to scrape application Prometheus metrics, such as those available at the standard /metrics endpoint on Pods.

To add a scrape job targeting all /metrics endpoints on your cluster Pods, complete the following steps:

  1. Add the following to the end of your Agent scrape config:

    . . .
    - job_name: "kubernetes-pods"
          - role: pod
          # Example relabel to scrape only Pods that have
          # " = true" annotation.
          #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
          #    action: keep
          #    regex: true
          # Example relabel to customize metric path based on Pod
          # " = <metric path>" annotation.
          #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
          #    action: replace
          #    target_label: __metrics_path__
          #    regex: (.+)
          # Example relabel to scrape only single, desired port for the Pod
          # based on Pod " = <port>" annotation.
          #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
          #    action: replace
          #    regex: ([^:]+)(?::\d+)?;(\d+)
          #    replacement: $1:$2
          #    target_label: __address__
          # Expose Pod labels as metric labels
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          # Expose Pod namespace as metric namespace label
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: namespace
          # Expose Pod name as metric name label
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: pod

    This config adds every defined Pod container port to the Agent’s scrape targets, which are discovered using Agent’s Kubernetes service discovery mechanism. You can optionally do any of the following:

    • Uncomment the relevant sections to customize the metrics path (the default is /metrics).
    • Specify a sample port.
    • Use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests.

    To learn more, refer to the Prometheus examples.

    For more about configuring Agent, refer to Configure Grafana Agent static mode. For more about available kubernetes_sd_configs labels and roles (we used the pod role here), refer to kubernetes_sd_config.

  2. Deploy the updated config into your Cluster using kubectl apply -f:

    kubectl apply -f <YOUR-CONFIGMAP>.yaml
  3. Restart the agent to pick up the config changes.

    kubectl rollout restart deployment/grafana-agent

For a complete example, refer to Monitor an app on Kubernetes using Grafana Agent.