Grafana Agent Flow mode Reference Components reference prometheus.operator.podmonitors


BETA: This is a beta component. Beta components are subject to breaking changes, and may be replaced with equivalent functionality that cover the same use case.

prometheus.operator.podmonitors discovers PodMonitor resources in your kubernetes cluster and scrapes the targets they reference. This component performs three main functions:

  1. Discover PodMonitor resources from your Kubernetes cluster.
  2. Discover Pods in your cluster that match those PodMonitors.
  3. Scrape metrics from those Pods, and forward them to a receiver.

The default configuration assumes the agent is running inside a Kubernetes cluster, and uses the in-cluster config to access the Kubernetes API. It can be run from outside the cluster by supplying connection info in the client block, but network level access to pods is required to scrape metrics from them.

PodMonitors may reference secrets for authenticating to targets to scrape them. In these cases, the secrets are loaded and refreshed only when the PodMonitor is updated or when this component refreshes its’ internal state, which happens on a 5-minute refresh cycle.


prometheus.operator.podmonitors "LABEL" {
    forward_to = RECEIVER_LIST


The following arguments are supported:

forward_tolist(MetricsReceiver)List of receivers to send scraped metrics to.yes
namespaceslist(string)List of namespaces to search for PodMonitor resources. If not specified, all namespaces will be


The following blocks are supported inside the definition of prometheus.operator.podmonitors:

clientclientConfigures Kubernetes client used to find
client > basic_authbasic_authConfigure basic authentication to the Kubernetes
client > authorizationauthorizationConfigure generic authorization to the Kubernetes
client > oauth2oauth2Configure OAuth2 for authenticating to the Kubernetes
client > oauth2 > tls_configtls_configConfigure TLS settings for connecting to the Kubernetes
client > tls_configtls_configConfigure TLS settings for connecting to the Kubernetes
ruleruleRelabeling rules to apply to discovered
selectorselectorLabel selector for which PodMonitors to
selector > match_expressionmatch_expressionLabel selector expression for which PodMonitors to
clusteringclusteringConfigure the component for when the Agent is running in clustered

The > symbol indicates deeper levels of nesting. For example, client > basic_auth refers to a basic_auth block defined inside a client block.

client block

The client block configures the Kubernetes client used to discover PodMonitors. If the client block isn’t provided, the default in-cluster configuration with the service account of the running Grafana Agent pod is used.

The following arguments are supported:

api_serverstringURL of the Kubernetes API
kubeconfig_filestringPath of the kubeconfig file to use for connecting to
bearer_token_filestringFile containing a bearer token to authenticate
proxy_urlstringHTTP proxy to proxy requests
follow_redirectsboolWhether redirects returned by the server should be followed.trueno
enable_http2boolWhether HTTP2 is supported for requests.trueno

At most one of the following can be provided:

basic_auth block

usernamestringBasic auth
passwordsecretBasic auth
password_filestringFile containing the basic auth

password and password_file are mutually exclusive and only one can be provided inside of a basic_auth block.

authorization block

typestringAuthorization type, for example, “Bearer”.no
credentials_filestringFile containing the secret

credential and credentials_file are mutually exclusive and only one can be provided inside of an authorization block.

oauth2 block

client_idstringOAuth2 client
client_secretsecretOAuth2 client
client_secret_filestringFile containing the OAuth2 client
scopeslist(string)List of scopes to authenticate
token_urlstringURL to fetch the token
endpoint_paramsmap(string)Optional parameters to append to the token
proxy_urlstringOptional proxy URL for OAuth2

client_secret and client_secret_file are mutually exclusive and only one can be provided inside of an oauth2 block.

The oauth2 block may also contain its own separate tls_config sub-block.

tls_config block

ca_pemstringCA PEM-encoded text to validate the server
ca_filestringCA certificate to validate the server
cert_pemstringCertificate PEM-encoded text for client
cert_filestringCertificate file for client
key_pemsecretKey PEM-encoded text for client
key_filestringKey file for client
server_namestringServerName extension to indicate the name of the
insecure_skip_verifyboolDisables validation of the server
min_versionstringMinimum acceptable TLS

The following pairs of arguments are mutually exclusive and cannot both be set simultaneously:

  • ca_pem and ca_file
  • cert_pem and cert_file
  • key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

When min_version is not provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If min_version is provided, it must be set to one of the following strings:

  • "TLS10" (TLS 1.0)
  • "TLS11" (TLS 1.1)
  • "TLS12" (TLS 1.2)
  • "TLS13" (TLS 1.3)

rule block

The rule block contains the definition of any relabeling rules that can be applied to an input metric. If more than one rule block is defined, the transformations are applied in top-down order.

The following arguments can be used to configure a rule. All arguments are optional. Omitted fields take their default values.

source_labelslist(string)The list of labels whose values are to be selected. Their content is concatenated using the separator and matched against
separatorstringThe separator used to concatenate the values present in source_labels.;no
regexstringA valid RE2 expression with support for parenthesized capture groups. Used to match the extracted value from the combination of the source_label and separator fields or filter labels during the labelkeep/labeldrop/labelmap actions.(.*)no
modulusuintA positive integer used to calculate the modulus of the hashed source label
target_labelstringLabel to which the resulting value will be written
replacementstringThe value against which a regex replace is performed, if the regex matches the extracted value. Supports previously captured groups.$1no
actionstringThe relabeling action to perform.replaceno

Here’s a list of the available actions, along with a brief description of their usage.

  • replace - Matches regex to the concatenated labels. If there’s a match, it replaces the content of the target_label using the contents of the replacement field.
  • keep - Keeps metrics where regex matches the string extracted using the source_labels and separator.
  • drop - Drops metrics where regex matches the string extracted using the source_labels and separator.
  • hashmod - Hashes the concatenated labels, calculates its modulo modulus and writes the result to the target_label.
  • labelmap - Matches regex against all label names. Any labels that match are renamed according to the contents of the replacement field.
  • labeldrop - Matches regex against all label names. Any labels that match are removed from the metric’s label set.
  • labelkeep - Matches regex against all label names. Any labels that don’t match are removed from the metric’s label set.
  • keepequal - Drop targets for which the concatenated source_labels do not match target_label.
  • dropequal - Drop targets for which the concatenated source_labels do match target_label.
  • lowercase - Sets target_label to the lowercase form of the concatenated source_labels.
  • uppercase - Sets target_label to the uppercase form of the concatenated source_labels.

Finally, note that the regex capture groups can be referred to using either the $CAPTURE_GROUP_NUMBER or ${CAPTURE_GROUP_NUMBER} notation.

selector block

The selector block describes a Kubernetes label selector for PodMonitors.

The following arguments are supported:

match_labelsmap(string)Label keys and values used to discover resources.{}no

When the match_labels argument is empty, all PodMonitor resources will be matched.

match_expression block

The match_expression block describes a Kubernetes label matcher expression for PodMonitors discovery.

The following arguments are supported:

keystringThe label name to match against.yes
operatorstringThe operator to use when matching.yes
valueslist(string)The values used when

The operator argument must be one of the following strings:

  • "In"
  • "NotIn"
  • "Exists"
  • "DoesNotExist"

If there are multiple match_expressions blocks inside of a selector block, they are combined together with AND clauses.

clustering (beta)

enabledboolEnables sharing targets with other cluster nodes.falseyes

When the agent is using clustering, and enabled is set to true, then this component instance opts-in to participating in the cluster to distribute scrape load between all cluster nodes.

Clustering assumes that all cluster nodes are running with the same configuration file, and that all prometheus.operator.podmonitors components that have opted-in to using clustering, over the course of a scrape interval have the same configuration.

All prometheus.operator.podmonitors components instances opting in to clustering use target labels and a consistent hashing algorithm to determine ownership for each of the targets between the cluster peers. Then, each peer only scrapes the subset of targets that it is responsible for, so that the scrape load is distributed. When a node joins or leaves the cluster, every peer recalculates ownership and continues scraping with the new target set. This performs better than hashmod sharding where all nodes have to be re-distributed, as only 1/N of the target’s ownership is transferred, but is eventually consistent (rather than fully consistent like hashmod sharding is).

If the agent is not running in clustered mode, then the block is a no-op, and prometheus.operator.podmonitors scrapes every target it receives in its arguments.

Exported fields

prometheus.operator.podmonitors does not export any fields. It forwards all metrics it scrapes to the receiver configures with the forward_to argument.

Component health

prometheus.operator.podmonitors is reported as unhealthy when given an invalid configuration, Prometheus components fail to initialize, or the connection to the Kubernetes API could not be established properly.

Debug information

prometheus.operator.podmonitors reports the status of the last scrape for each configured scrape job on the component’s debug endpoint, including discovered labels, and the last scrape time.

It also exposes some debug information for each PodMonitor it has discovered, including any errors found while reconciling the scrape configuration from the PodMonitor.

Debug metrics


This example discovers all PodMonitors in your cluster, and forwards collected metrics to a prometheus.remote_write component.

prometheus.remote_write "staging" {
  // Send metrics to a locally running Mimir.
  endpoint {
    url = "http://mimir:9009/api/v1/push"

    basic_auth {
      username = "example-user"
      password = "example-password"

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]

This example will limit discovered PodMonitors to ones with the label team=ops in a specific namespace: my-app.

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
    namespaces = ["my-app"]
    selector {
        match_expression {
            key = "team"
            operator = "In"
            values = ["ops"]

This example will apply additional relabel rules to discovered targets to filter by hostname. This may be useful if running the agent as a DaemonSet.

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
    rule {
      action = "keep"
      regex = env("HOSTNAME")
      source_labels = ["__meta_kubernetes_pod_node_name"]