---
title: "prometheus.operator.probes | Grafana Cloud documentation"
description: "Learn about prometheus.operator.probes"
---

# `prometheus.operator.probes`

`prometheus.operator.probes` discovers [Probe](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.Probe) resources in your Kubernetes cluster and scrapes the targets they reference. This component performs three main functions:

1. Discover Probe resources from your Kubernetes cluster.
2. Discover targets or ingresses that match those Probes.
3. Scrape metrics from those endpoints, and forward them to a receiver.

The default configuration assumes Alloy is running inside a Kubernetes cluster, and uses the in-cluster configuration to access the Kubernetes API. You can run it from outside the cluster by supplying connection info in the `client` block, but network level access to pods is required to scrape metrics from them.

Probes may reference secrets for authenticating to targets to scrape them. In these cases, the secrets are loaded and refreshed only when the Probe is updated or when this component refreshes its’ internal state, which happens on a 5-minute refresh cycle.

## Usage

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.operator.probes "<LABEL>" {
    forward_to = <RECEIVER_LIST>
}
```

## Arguments

You can use the following arguments with `prometheus.operator.probes`:

Expand table

| Name                    | Type                    | Description                                                                                      | Default | Required |
|-------------------------|-------------------------|--------------------------------------------------------------------------------------------------|---------|----------|
| `forward_to`            | `list(MetricsReceiver)` | List of receivers to send scraped metrics to.                                                    |         | yes      |
| `informer_sync_timeout` | `duration`              | Timeout for initial sync of Probe resources.                                                     | `"1m"`  | no       |
| `namespaces`            | `list(string)`          | List of namespaces to search for Probe resources. If not specified, all namespaces are searched. |         | no       |

## Blocks

You can use the following blocks with `prometheus.operator.probes`:

No valid configuration blocks found.

### `client`

The `client` block configures the Kubernetes client used to discover Probes. If the `client` block isn’t provided, the default in-cluster configuration with the service account of the running Alloy Pod is used.

The following arguments are supported:

Expand table

| Name                     | Type                | Description                                                                                      | Default | Required |
|--------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------|----------|
| `api_server`             | `string`            | URL of the Kubernetes API server.                                                                |         | no       |
| `bearer_token_file`      | `string`            | File containing a bearer token to authenticate with.                                             |         | no       |
| `bearer_token`           | `secret`            | Bearer token to authenticate with.                                                               |         | no       |
| `enable_http2`           | `bool`              | Whether HTTP2 is supported for requests.                                                         | `true`  | no       |
| `follow_redirects`       | `bool`              | Whether redirects returned by the server should be followed.                                     | `true`  | no       |
| `http_headers`           | `map(list(secret))` | Custom HTTP headers to be sent along with each request. The map key is the header name.          |         | no       |
| `kubeconfig_file`        | `string`            | Path of the `kubeconfig` file to use for connecting to Kubernetes.                               |         | no       |
| `no_proxy`               | `string`            | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. |         | no       |
| `proxy_connect_header`   | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests.                                    |         | no       |
| `proxy_from_environment` | `bool`              | Use the proxy URL indicated by environment variables.                                            | `false` | no       |
| `proxy_url`              | `string`            | HTTP proxy to send requests through.                                                             |         | no       |

At most, one of the following can be provided:

- \[`authorization`]\[authorization] block
- \[`basic_auth`]\[basic\_auth] block
- \[`bearer_token_file`]\[client] argument
- \[`bearer_token`]\[client] argument
- \[`oauth2`]\[oauth2] block

`no_proxy` can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. `proxy_url` must be configured if `no_proxy` is configured.

`proxy_from_environment` uses the environment variables HTTP\_PROXY, HTTPS\_PROXY, and NO\_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO\_PROXY. `proxy_url` and `no_proxy` must not be configured if `proxy_from_environment` is configured.

`proxy_connect_header` should only be configured if `proxy_url` or `proxy_from_environment` are configured.

### `authorization`

Expand table

| Name               | Type     | Description                                | Default | Required |
|--------------------|----------|--------------------------------------------|---------|----------|
| `credentials_file` | `string` | File containing the secret value.          |         | no       |
| `credentials`      | `secret` | Secret value.                              |         | no       |
| `type`             | `string` | Authorization type, for example, “Bearer”. |         | no       |

`credential` and `credentials_file` are mutually exclusive, and only one can be provided inside an `authorization` block.

> Warning
> 
> Using `credentials_file` causes the file to be read on every outgoing request. Use the `local.file` component with the `credentials` attribute instead to avoid unnecessary reads.

### `basic_auth`

Expand table

| Name            | Type     | Description                              | Default | Required |
|-----------------|----------|------------------------------------------|---------|----------|
| `password_file` | `string` | File containing the basic auth password. |         | no       |
| `password`      | `secret` | Basic auth password.                     |         | no       |
| `username`      | `string` | Basic auth username.                     |         | no       |

`password` and `password_file` are mutually exclusive, and only one can be provided inside a `basic_auth` block.

> Warning
> 
> Using `password_file` causes the file to be read on every outgoing request. Use the `local.file` component with the `password` attribute instead to avoid unnecessary reads.

### `oauth2`

Expand table

| Name                     | Type                | Description                                                                                      | Default | Required |
|--------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------|----------|
| `client_id`              | `string`            | OAuth2 client ID.                                                                                |         | no       |
| `client_secret_file`     | `string`            | File containing the OAuth2 client secret.                                                        |         | no       |
| `client_secret`          | `secret`            | OAuth2 client secret.                                                                            |         | no       |
| `endpoint_params`        | `map(string)`       | Optional parameters to append to the token URL.                                                  |         | no       |
| `no_proxy`               | `string`            | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. |         | no       |
| `proxy_connect_header`   | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests.                                    |         | no       |
| `proxy_from_environment` | `bool`              | Use the proxy URL indicated by environment variables.                                            | `false` | no       |
| `proxy_url`              | `string`            | HTTP proxy to send requests through.                                                             |         | no       |
| `scopes`                 | `list(string)`      | List of scopes to authenticate with.                                                             |         | no       |
| `token_url`              | `string`            | URL to fetch the token from.                                                                     |         | no       |

`client_secret` and `client_secret_file` are mutually exclusive, and only one can be provided inside an `oauth2` block.

> Warning
> 
> Using `client_secret_file` causes the file to be read on every outgoing request. Use the `local.file` component with the `client_secret` attribute instead to avoid unnecessary reads.

The `oauth2` block may also contain a separate `tls_config` sub-block.

`no_proxy` can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. `proxy_url` must be configured if `no_proxy` is configured.

`proxy_from_environment` uses the environment variables HTTP\_PROXY, HTTPS\_PROXY, and NO\_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO\_PROXY. `proxy_url` and `no_proxy` must not be configured if `proxy_from_environment` is configured.

`proxy_connect_header` should only be configured if `proxy_url` or `proxy_from_environment` are configured.

### `tls_config`

Expand table

| Name                   | Type     | Description                                              | Default | Required |
|------------------------|----------|----------------------------------------------------------|---------|----------|
| `ca_pem`               | `string` | CA PEM-encoded text to validate the server with.         |         | no       |
| `ca_file`              | `string` | CA certificate to validate the server with.              |         | no       |
| `cert_pem`             | `string` | Certificate PEM-encoded text for client authentication.  |         | no       |
| `cert_file`            | `string` | Certificate file for client authentication.              |         | no       |
| `insecure_skip_verify` | `bool`   | Disables validation of the server certificate.           |         | no       |
| `key_file`             | `string` | Key file for client authentication.                      |         | no       |
| `key_pem`              | `secret` | Key PEM-encoded text for client authentication.          |         | no       |
| `min_version`          | `string` | Minimum acceptable TLS version.                          |         | no       |
| `server_name`          | `string` | ServerName extension to indicate the name of the server. |         | no       |

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

- `ca_pem` and `ca_file`
- `cert_pem` and `cert_file`
- `key_pem` and `key_file`

When configuring client authentication, both the client certificate (using `cert_pem` or `cert_file`) and the client key (using `key_pem` or `key_file`) must be provided.

When `min_version` isn’t provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If `min_version` is provided, it must be set to one of the following strings:

- `"TLS10"` (TLS 1.0)
- `"TLS11"` (TLS 1.1)
- `"TLS12"` (TLS 1.2)
- `"TLS13"` (TLS 1.3)

### `clustering`

Expand table

| Name      | Type   | Description                                       | Default | Required |
|-----------|--------|---------------------------------------------------|---------|----------|
| `enabled` | `bool` | Enables sharing targets with other cluster nodes. | `false` | yes      |

When Alloy is running in [clustered mode](/docs/grafana-cloud/send-data/alloy/reference/cli/run/#clustering), and `enabled` is set to true, then this component instance opts-in to participating in the cluster to distribute scrape load between all cluster nodes.

Clustering assumes that all cluster nodes are running with the same configuration file, and that all `prometheus.operator.probes` components that have opted-in to using clustering, over the course of a scrape interval have the same configuration.

All `prometheus.operator.probes` components instances opting in to clustering use target labels and a consistent hashing algorithm to determine ownership for each of the targets between the cluster peers. Then, each peer only scrapes the subset of targets that it’s responsible for, so that the scrape load is distributed. When a node joins or leaves the cluster, every peer recalculates ownership and continues scraping with the new target set. This performs better than hashmod sharding where *all* nodes have to be re-distributed, as only 1/N of the target’s ownership is transferred, but is eventually consistent (rather than fully consistent like hashmod sharding is).

If Alloy is *not* running in clustered mode, then the block is a no-op, and `prometheus.operator.probes` scrapes every target it receives in its arguments.

### `rule`

The `rule` block contains the definition of any relabeling rules that can be applied to an input metric. If more than one `rule` block is defined, the transformations are applied in top-down order.

The following arguments can be used to configure a `rule`. All arguments are optional. Omitted fields take their default values.

Expand table

| Name            | Type           | Description                                                                                                                                                                                                                                   | Default | Required |
|-----------------|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|----------|
| `action`        | `string`       | The relabeling action to perform.                                                                                                                                                                                                             | replace | no       |
| `modulus`       | `uint`         | A positive integer used to calculate the modulus of the hashed source label values.                                                                                                                                                           |         | no       |
| `regex`         | `string`       | A valid RE2 expression with support for parenthesized capture groups. Used to match the extracted value from the combination of the `source_label` and `separator` fields or filter labels during the `labelkeep/labeldrop/labelmap` actions. | `(.*)`  | no       |
| `replacement`   | `string`       | The value against which a regular expression replace is performed, if the regular expression matches the extracted value. Supports previously captured groups.                                                                                | `"$1"`  | no       |
| `separator`     | `string`       | The separator used to concatenate the values present in `source_labels`.                                                                                                                                                                      | ;       | no       |
| `source_labels` | `list(string)` | The list of labels whose values are to be selected. Their content is concatenated using the `separator` and matched against `regex`.                                                                                                          |         | no       |
| `target_label`  | `string`       | Label to which the resulting value will be written to.                                                                                                                                                                                        |         | no       |

You can use the following actions:

- `drop` - Drops metrics where `regex` matches the string extracted using the `source_labels` and `separator`.
- `dropequal` - Drop targets for which the concatenated `source_labels` do match `target_label`.
- `hashmod` - Hashes the concatenated labels, calculates its modulo `modulus` and writes the result to the `target_label`.
- `keep` - Keeps metrics where `regex` matches the string extracted using the `source_labels` and `separator`.
- `keepequal` - Drop targets for which the concatenated `source_labels` don’t match `target_label`.
- `labeldrop` - Matches `regex` against all label names. Any labels that match are removed from the metric’s label set.
- `labelkeep` - Matches `regex` against all label names. Any labels that don’t match are removed from the metric’s label set.
- `labelmap` - Matches `regex` against all label names. Any labels that match are renamed according to the contents of the `replacement` field.
- `lowercase` - Sets `target_label` to the lowercase form of the concatenated `source_labels`.
- `replace` - Matches `regex` to the concatenated labels. If there’s a match, it replaces the content of the `target_label` using the contents of the `replacement` field.
- `uppercase` - Sets `target_label` to the uppercase form of the concatenated `source_labels`.

> Note
> 
> The regular expression capture groups can be referred to using either the `$CAPTURE_GROUP_NUMBER` or `${CAPTURE_GROUP_NUMBER}` notation.

### `scrape`

Expand table

| Name                          | Type       | Description                                                                                                                  | Default | Required |
|-------------------------------|------------|------------------------------------------------------------------------------------------------------------------------------|---------|----------|
| `default_sample_limit`        | `int`      | The default maximum samples per scrape. Used as the default if the target resource doesn’t provide a sample limit.           |         | no       |
| `default_scrape_interval`     | `duration` | The default interval between scraping targets. Used as the default if the target resource doesn’t provide a scrape interval. | `1m`    | no       |
| `default_scrape_timeout`      | `duration` | The default timeout for scrape requests. Used as the default if the target resource doesn’t provide a scrape timeout.        | `10s`   | no       |
| `enable_type_and_unit_labels` | `bool`     | (Experimental) Whether the metric type and unit should be added as labels to scraped metrics.                                | `false` | no       |
| `honor_metadata`              | `bool`     | (Experimental) Indicates whether to send metric metadata to downstream components.                                           | `false` | no       |
| `scrape_native_histograms`    | `bool`     | Whether to scrape native histograms from targets.                                                                            | `false` | no       |

> **EXPERIMENTAL**: The `honor_metadata` and `enable_type_and_unit_labels` arguments are [experimental](/docs/release-life-cycle/) features.
> 
> If you enable the `honor_metadata` argument, resource consumption may increase, particularly if you ingest many metrics with different names. Some downstream components aren’t compatible with Prometheus metadata. The following components are compatible:
> 
> - `otelcol.receiver.prometheus`
> - `prometheus.remote_write` only when configured for Remote Write v2.
> - `prometheus.write_queue`
> 
> When `enable_type_and_unit_labels` argument is enabled and available from the scrape, the metric type and unit are added as labels to each scraped sample. This provides additional schema information about metrics directly in the label set. This feature doesn’t require downstream components to support Remote Write v2.
> 
> Experimental features are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental feature, you must set the `stability.level` [flag](/docs/alloy/latest/reference/cli/run/) to `experimental`.

### `selector`

The `selector` block describes a Kubernetes label selector for Probes.

The following arguments are supported:

Expand table

| Name           | Type          | Description                                       | Default | Required |
|----------------|---------------|---------------------------------------------------|---------|----------|
| `match_labels` | `map(string)` | Label keys and values used to discover resources. | `{}`    | no       |

When the `match_labels` argument is empty, all Probe resources are matched.

### `match_expression`

The `match_expression` block describes a Kubernetes label matcher expression for Probes discovery.

The following arguments are supported:

Expand table

| Name       | Type           | Description                        | Default | Required |
|------------|----------------|------------------------------------|---------|----------|
| `key`      | `string`       | The label name to match against.   |         | yes      |
| `operator` | `string`       | The operator to use when matching. |         | yes      |
| `values`   | `list(string)` | The values used when matching.     |         | no       |

The `operator` argument must be one of the following strings:

- `"In"`
- `"NotIn"`
- `"Exists"`
- `"DoesNotExist"`

If there are multiple `match_expressions` blocks inside of a `selector` block, they’re combined together with AND clauses.

## Exported fields

`prometheus.operator.probes` doesn’t export any fields. It forwards all metrics it scrapes to the receivers configured with the `forward_to` argument.

## Component health

`prometheus.operator.probes` is reported as unhealthy when given an invalid configuration, Prometheus components fail to initialize, or the connection to the Kubernetes API couldn’t be established properly.

## Debug information

`prometheus.operator.probes` reports the status of the last scrape for each configured scrape job on the component’s debug endpoint, including discovered labels, and the last scrape time.

It also exposes some debug information for each Probe it has discovered, including any errors found while reconciling the scrape configuration from the Probe.

## Debug metrics

`prometheus.operator.probes` doesn’t expose any component-specific debug metrics.

## Example

The following example discovers all Probes in your cluster, and forwards collected metrics to a `prometheus.remote_write` component.

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.remote_write "staging" {
  // Send metrics to a locally running Mimir.
  endpoint {
    url = "http://mimir:9009/api/v1/push"

    basic_auth {
      username = "example-user"
      password = "example-password"
    }
  }
}

prometheus.operator.probes "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
}
```

The following example limits discovered Probes to ones with the label `team=ops` in a specific namespace: `my-app`.

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.operator.probes "pods" {
    forward_to = [prometheus.remote_write.staging.receiver]
    namespaces = ["my-app"]
    selector {
        match_expression {
            key = "team"
            operator = "In"
            values = ["ops"]
        }
    }
}
```

The following example applies additional relabel rules to discovered targets to filter by hostname. This may be useful if running Alloy as a DaemonSet.

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.operator.probes "probes" {
    forward_to = [prometheus.remote_write.staging.receiver]
    rule {
      action = "keep"
      regex = sys.env("HOSTNAME")
      source_labels = ["__meta_kubernetes_pod_node_name"]
    }
}
```

## Compatible components

`prometheus.operator.probes` can accept arguments from the following components:

- Components that export [Prometheus `MetricsReceiver`](/docs/grafana-cloud/send-data/alloy/reference/compatibility/#prometheus-metricsreceiver-exporters)

> Note
> 
> Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.