Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Alloy Reference Components prometheus prometheus.operator.podmonitors

prometheus.operator.podmonitors

prometheus.operator.podmonitors discovers PodMonitor resources in your Kubernetes cluster and scrapes the targets they reference. This component performs three main functions:

1. Discover PodMonitor resources from your Kubernetes cluster.
2. Discover Pods in your cluster that match those PodMonitors.
3. Scrape metrics from those Pods, and forward them to a receiver.

The default configuration assumes Alloy is running inside a Kubernetes cluster, and uses the in-cluster configuration to access the Kubernetes API. You can run it from outside the cluster by supplying connection info in the client block, but network level access to Pods is required to scrape metrics from them.

PodMonitors may reference secrets for authenticating to targets to scrape them. In these cases, the secrets are loaded and refreshed only when the PodMonitor is updated or when this component refreshes its’ internal state, which happens on a 5-minute refresh cycle.

Usage

alloy Copy

prometheus.operator.podmonitors "<LABEL>" {
    forward_to = <RECEIVER_LIST>
}

Arguments

You can use the following arguments with prometheus.operator.podmonitors:

Blocks

You can use the following blocks with prometheus.operator.podmonitors:

The > symbol indicates deeper levels of nesting. For example, client > basic_auth refers to a basic_auth block defined inside a client block.

client

The client block configures the Kubernetes client used to discover PodMonitors. If the client block isn’t provided, the default in-cluster configuration with the service account of the running Alloy Pod is used.

The following arguments are supported:

At most, one of the following can be provided:

• authorization block
• basic_auth block
• bearer_token_file argument
• bearer_token argument
• oauth2 block

authorization

basic_auth

oauth2

tls_config

clustering

When Alloy is using clustering, and enabled is set to true, then this component instance opts-in to participating in the cluster to distribute scrape load between all cluster nodes.

Clustering assumes that all cluster nodes are running with the same configuration file, and that all prometheus.operator.podmonitors components that have opted-in to using clustering, over the course of a scrape interval have the same configuration.

All prometheus.operator.podmonitors components instances opting in to clustering use target labels and a consistent hashing algorithm to determine ownership for each of the targets between the cluster peers. Then, each peer only scrapes the subset of targets that it’s responsible for, so that the scrape load is distributed. When a node joins or leaves the cluster, every peer recalculates ownership and continues scraping with the new target set. This performs better than hashmod sharding where all nodes have to be re-distributed, as only 1/N of the target’s ownership is transferred, but is eventually consistent (rather than fully consistent like hashmod sharding is).

If Alloy is not running in clustered mode, then the block is a no-op, and prometheus.operator.podmonitors scrapes every target it receives in its arguments.

rule

scrape

selector

The selector block describes a Kubernetes label selector for PodMonitors.

The following arguments are supported:

When the match_labels argument is empty, all PodMonitor resources are matched.

match_expression

The match_expression block describes a Kubernetes label matcher expression for PodMonitors discovery.

The following arguments are supported:

The operator argument must be one of the following strings:

• "In"
• "NotIn"
• "Exists"
• "DoesNotExist"

If there are multiple match_expressions blocks inside of a selector block, they are combined together with AND clauses.

Exported fields

prometheus.operator.podmonitors doesn’t export any fields. It forwards all metrics it scrapes to the receiver configures with the forward_to argument.

Component health

prometheus.operator.podmonitors is reported as unhealthy when given an invalid configuration, Prometheus components fail to initialize, or the connection to the Kubernetes API couldn’t be established properly.

Debug information

prometheus.operator.podmonitors reports the status of the last scrape for each configured scrape job on the component’s debug endpoint, including discovered labels, and the last scrape time.

It also exposes some debug information for each PodMonitor it has discovered, including any errors found while reconciling the scrape configuration from the PodMonitor.

Debug metrics

prometheus.operator.podmonitors doesn’t expose any component-specific debug metrics.

Example

The following example discovers all PodMonitors in your cluster, and forwards collected metrics to a prometheus.remote_write component.

alloy Copy

prometheus.remote_write "staging" {
  // Send metrics to a locally running Mimir.
  endpoint {
    url = "http://mimir:9009/api/v1/push"

    basic_auth {
      username = "example-user"
      password = "example-password"
    }
  }
}

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
}

This example will limit discovered PodMonitors to ones with the label team=ops in a specific namespace: my-app.

alloy Copy

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
    namespaces = ["my-app"]
    selector {
        match_expression {
            key = "team"
            operator = "In"
            values = ["ops"]
        }
    }
}

The following example applies additional relabel rules to discovered targets to filter by hostname. This may be useful if running Alloy as a DaemonSet.

alloy Copy

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
    rule {
      action = "keep"
      regex = sys.env("<HOSTNAME>")
      source_labels = ["__meta_kubernetes_pod_node_name"]
    }
}

Compatible components

prometheus.operator.podmonitors can accept arguments from the following components:

• Components that export Prometheus MetricsReceiver