Documentation Grafana Cloud Instrument and send data Grafana Alloy Reference Components prometheus prometheus.operator.probes

prometheus.operator.probes

prometheus.operator.probes discovers Probe resources in your Kubernetes cluster and scrapes the targets they reference. This component performs three main functions:

1. Discover Probe resources from your Kubernetes cluster.
2. Discover targets or ingresses that match those Probes.
3. Scrape metrics from those endpoints, and forward them to a receiver.

The default configuration assumes Alloy is running inside a Kubernetes cluster, and uses the in-cluster config to access the Kubernetes API. It can be run from outside the cluster by supplying connection info in the client block, but network level access to pods is required to scrape metrics from them.

Probes may reference secrets for authenticating to targets to scrape them. In these cases, the secrets are loaded and refreshed only when the Probe is updated or when this component refreshes its’ internal state, which happens on a 5-minute refresh cycle.

Usage

alloy Copy

prometheus.operator.probes "LABEL" {
    forward_to = RECEIVER_LIST
}

Arguments

The following arguments are supported:

Blocks

The following blocks are supported inside the definition of prometheus.operator.probes:

The > symbol indicates deeper levels of nesting. For example, client > basic_auth refers to a basic_auth block defined inside a client block.

client block

The client block configures the Kubernetes client used to discover Probes. If the client block isn’t provided, the default in-cluster configuration with the service account of the running Alloy pod is used.

The following arguments are supported:

At most, one of the following can be provided:

• bearer_token argument.
• bearer_token_file argument.
• basic_auth block.
• authorization block.
• oauth2 block.

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_from_environment uses the environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO_PROXY. proxy_url and no_proxy must not be configured if proxy_from_environment is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

basic_auth block

password and password_file are mutually exclusive, and only one can be provided inside a basic_auth block.

authorization block

credential and credentials_file are mutually exclusive, and only one can be provided inside an authorization block.

oauth2 block

client_secret and client_secret_file are mutually exclusive, and only one can be provided inside an oauth2 block.

The oauth2 block may also contain a separate tls_config sub-block.

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_from_environment uses the environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO_PROXY. proxy_url and no_proxy must not be configured if proxy_from_environment is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

tls_config block

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

• ca_pem and ca_file
• cert_pem and cert_file
• key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

When min_version isn’t provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If min_version is provided, it must be set to one of the following strings:

• "TLS10" (TLS 1.0)
• "TLS11" (TLS 1.1)
• "TLS12" (TLS 1.2)
• "TLS13" (TLS 1.3)

rule block

The rule block contains the definition of any relabeling rules that can be applied to an input metric. If more than one rule block is defined, the transformations are applied in top-down order.

The following arguments can be used to configure a rule. All arguments are optional. Omitted fields take their default values.

You can use the following actions:

• drop - Drops metrics where regex matches the string extracted using the source_labels and separator.
• dropequal - Drop targets for which the concatenated source_labels do match target_label.
• hashmod - Hashes the concatenated labels, calculates its modulo modulus and writes the result to the target_label.
• keep - Keeps metrics where regex matches the string extracted using the source_labels and separator.
• keepequal - Drop targets for which the concatenated source_labels don’t match target_label.
• labeldrop - Matches regex against all label names. Any labels that match are removed from the metric’s label set.
• labelkeep - Matches regex against all label names. Any labels that don’t match are removed from the metric’s label set.
• labelmap - Matches regex against all label names. Any labels that match are renamed according to the contents of the replacement field.
• lowercase - Sets target_label to the lowercase form of the concatenated source_labels.
• replace - Matches regex to the concatenated labels. If there’s a match, it replaces the content of the target_label using the contents of the replacement field.
• uppercase - Sets target_label to the uppercase form of the concatenated source_labels.

scrape block

selector block

The selector block describes a Kubernetes label selector for Probes.

The following arguments are supported:

When the match_labels argument is empty, all Probe resources will be matched.

match_expression block

The match_expression block describes a Kubernetes label matcher expression for Probes discovery.

The following arguments are supported:

The operator argument must be one of the following strings:

• "In"
• "NotIn"
• "Exists"
• "DoesNotExist"

If there are multiple match_expressions blocks inside of a selector block, they are combined together with AND clauses.

clustering block

When Alloy is running in clustered mode, and enabled is set to true, then this component instance opts-in to participating in the cluster to distribute scrape load between all cluster nodes.

Clustering assumes that all cluster nodes are running with the same configuration file, and that all prometheus.operator.probes components that have opted-in to using clustering, over the course of a scrape interval have the same configuration.

All prometheus.operator.probes components instances opting in to clustering use target labels and a consistent hashing algorithm to determine ownership for each of the targets between the cluster peers. Then, each peer only scrapes the subset of targets that it is responsible for, so that the scrape load is distributed. When a node joins or leaves the cluster, every peer recalculates ownership and continues scraping with the new target set. This performs better than hashmod sharding where all nodes have to be re-distributed, as only 1/N of the target’s ownership is transferred, but is eventually consistent (rather than fully consistent like hashmod sharding is).

If Alloy is not running in clustered mode, then the block is a no-op, and prometheus.operator.probes scrapes every target it receives in its arguments.

Exported fields

prometheus.operator.probes does not export any fields. It forwards all metrics it scrapes to the receivers configured with the forward_to argument.

Component health

prometheus.operator.probes is reported as unhealthy when given an invalid configuration, Prometheus components fail to initialize, or the connection to the Kubernetes API could not be established properly.

Debug information

prometheus.operator.probes reports the status of the last scrape for each configured scrape job on the component’s debug endpoint, including discovered labels, and the last scrape time.

It also exposes some debug information for each Probe it has discovered, including any errors found while reconciling the scrape configuration from the Probe.

Debug metrics

prometheus.operator.probes does not expose any component-specific debug metrics.

Example

This example discovers all Probes in your cluster, and forwards collected metrics to a prometheus.remote_write component.

alloy Copy

prometheus.remote_write "staging" {
  // Send metrics to a locally running Mimir.
  endpoint {
    url = "http://mimir:9009/api/v1/push"

    basic_auth {
      username = "example-user"
      password = "example-password"
    }
  }
}

prometheus.operator.probes "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
}

This example will limit discovered Probes to ones with the label team=ops in a specific namespace: my-app.

alloy Copy

prometheus.operator.probes "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
    namespaces = ["my-app"]
    selector {
        match_expression {
            key = "team"
            operator = "In"
            values = ["ops"]
        }
    }
}

This example will apply additional relabel rules to discovered targets to filter by hostname. This may be useful if running Alloy as a DaemonSet.

alloy Copy

prometheus.operator.probes "probes" {
    forward_to = [prometheus.remote_write.staging.receiver]
    rule {
      action = "keep"
      regex = sys.env("HOSTNAME")
      source_labels = ["__meta_kubernetes_pod_node_name"]
    }
}

Compatible components

prometheus.operator.probes can accept arguments from the following components:

• Components that export Prometheus MetricsReceiver