Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.
Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
prometheus.operator.servicemonitors
BETA: This is a beta component. Beta components are subject to breaking changes, and may be replaced with equivalent functionality that cover the same use case.
prometheus.operator.servicemonitors
discovers ServiceMonitor resources in your kubernetes cluster and scrapes the targets they reference. This component performs three main functions:
- Discover ServiceMonitor resources from your Kubernetes cluster.
- Discover Services and Endpoints in your cluster that match those ServiceMonitors.
- Scrape metrics from those Endpoints, and forward them to a receiver.
The default configuration assumes the agent is running inside a Kubernetes cluster, and uses the in-cluster config to access the Kubernetes API. It can be run from outside the cluster by supplying connection info in the client
block, but network level access to discovered endpoints is required to scrape metrics from them.
ServiceMonitors may reference secrets for authenticating to targets to scrape them. In these cases, the secrets are loaded and refreshed only when the ServiceMonitor is updated or when this component refreshes its’ internal state, which happens on a 5-minute refresh cycle.
Usage
prometheus.operator.servicemonitors "LABEL" {
forward_to = RECEIVER_LIST
}
Arguments
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
forward_to | list(MetricsReceiver) | List of receivers to send scraped metrics to. | yes | |
namespaces | list(string) | List of namespaces to search for ServiceMonitor resources. If not specified, all namespaces will be searched. | no |
Blocks
The following blocks are supported inside the definition of prometheus.operator.servicemonitors
:
Hierarchy | Block | Description | Required |
---|---|---|---|
client | client | Configures Kubernetes client used to find ServiceMonitors. | no |
client > basic_auth | basic_auth | Configure basic authentication to the Kubernetes API. | no |
client > authorization | authorization | Configure generic authorization to the Kubernetes API. | no |
client > oauth2 | oauth2 | Configure OAuth2 for authenticating to the Kubernetes API. | no |
client > oauth2 > tls_config | tls_config | Configure TLS settings for connecting to the Kubernetes API. | no |
client > tls_config | tls_config | Configure TLS settings for connecting to the Kubernetes API. | no |
rule | rule | Relabeling rules to apply to discovered targets. | no |
selector | selector | Label selector for which ServiceMonitors to discover. | no |
selector > match_expression | match_expression | Label selector expression for which ServiceMonitors to discover. | no |
clustering | clustering | Configure the component for when the Agent is running in clustered mode. | no |
The >
symbol indicates deeper levels of nesting. For example, client > basic_auth
refers to a basic_auth
block defined
inside a client
block.
client block
The client
block configures the Kubernetes client used to discover ServiceMonitors. If the client
block isn’t provided, the default in-cluster
configuration with the service account of the running Grafana Agent pod is
used.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
api_server | string | URL of the Kubernetes API server. | no | |
kubeconfig_file | string | Path of the kubeconfig file to use for connecting to Kubernetes. | no | |
bearer_token_file | string | File containing a bearer token to authenticate with. | no | |
proxy_url | string | HTTP proxy to proxy requests through. | no | |
follow_redirects | bool | Whether redirects returned by the server should be followed. | true | no |
enable_http2 | bool | Whether HTTP2 is supported for requests. | true | no |
At most one of the following can be provided:
basic_auth block
Name | Type | Description | Default | Required |
---|---|---|---|---|
username | string | Basic auth username. | no | |
password | secret | Basic auth password. | no | |
password_file | string | File containing the basic auth password. | no |
password
and password_file
are mutually exclusive and only one can be
provided inside of a basic_auth
block.
authorization block
Name | Type | Description | Default | Required |
---|---|---|---|---|
type | string | Authorization type, for example, “Bearer”. | no | |
credentials | secret | Secret value. | no | |
credentials_file | string | File containing the secret value. | no |
credential
and credentials_file
are mutually exclusive and only one can be
provided inside of an authorization
block.
oauth2 block
Name | Type | Description | Default | Required |
---|---|---|---|---|
client_id | string | OAuth2 client ID. | no | |
client_secret | secret | OAuth2 client secret. | no | |
client_secret_file | string | File containing the OAuth2 client secret. | no | |
scopes | list(string) | List of scopes to authenticate with. | no | |
token_url | string | URL to fetch the token from. | no | |
endpoint_params | map(string) | Optional parameters to append to the token URL. | no | |
proxy_url | string | Optional proxy URL for OAuth2 requests. | no |
client_secret
and client_secret_file
are mutually exclusive and only one
can be provided inside of an oauth2
block.
The oauth2
block may also contain its own separate tls_config
sub-block.
tls_config block
Name | Type | Description | Default | Required |
---|---|---|---|---|
ca_pem | string | CA PEM-encoded text to validate the server with. | no | |
ca_file | string | CA certificate to validate the server with. | no | |
cert_pem | string | Certificate PEM-encoded text for client authentication. | no | |
cert_file | string | Certificate file for client authentication. | no | |
key_pem | secret | Key PEM-encoded text for client authentication. | no | |
key_file | string | Key file for client authentication. | no | |
server_name | string | ServerName extension to indicate the name of the server. | no | |
insecure_skip_verify | bool | Disables validation of the server certificate. | no | |
min_version | string | Minimum acceptable TLS version. | no |
The following pairs of arguments are mutually exclusive and cannot both be set simultaneously:
ca_pem
andca_file
cert_pem
andcert_file
key_pem
andkey_file
When configuring client authentication, both the client certificate (using
cert_pem
or cert_file
) and the client key (using key_pem
or key_file
)
must be provided.
When min_version
is not provided, the minimum acceptable TLS version is
inherited from Go’s default minimum version, TLS 1.2. If min_version
is
provided, it must be set to one of the following strings:
"TLS10"
(TLS 1.0)"TLS11"
(TLS 1.1)"TLS12"
(TLS 1.2)"TLS13"
(TLS 1.3)
rule block
The rule
block contains the definition of any relabeling rules that can be
applied to an input metric. If more than one rule
block is defined, the
transformations are applied in top-down order.
The following arguments can be used to configure a rule
. All arguments are
optional. Omitted fields take their default values.
Name | Type | Description | Default | Required |
---|---|---|---|---|
source_labels | list(string) | The list of labels whose values are to be selected. Their content is concatenated using the separator and matched against regex . | no | |
separator | string | The separator used to concatenate the values present in source_labels . | ; | no |
regex | string | A valid RE2 expression with support for parenthesized capture groups. Used to match the extracted value from the combination of the source_label and separator fields or filter labels during the labelkeep/labeldrop/labelmap actions. | (.*) | no |
modulus | uint | A positive integer used to calculate the modulus of the hashed source label values. | no | |
target_label | string | Label to which the resulting value will be written to. | no | |
replacement | string | The value against which a regex replace is performed, if the regex matches the extracted value. Supports previously captured groups. | $1 | no |
action | string | The relabeling action to perform. | replace | no |
Here’s a list of the available actions, along with a brief description of their usage.
replace
- Matchesregex
to the concatenated labels. If there’s a match, it replaces the content of thetarget_label
using the contents of thereplacement
field.keep
- Keeps metrics whereregex
matches the string extracted using thesource_labels
andseparator
.drop
- Drops metrics whereregex
matches the string extracted using thesource_labels
andseparator
.hashmod
- Hashes the concatenated labels, calculates its modulomodulus
and writes the result to thetarget_label
.labelmap
- Matchesregex
against all label names. Any labels that match are renamed according to the contents of thereplacement
field.labeldrop
- Matchesregex
against all label names. Any labels that match are removed from the metric’s label set.labelkeep
- Matchesregex
against all label names. Any labels that don’t match are removed from the metric’s label set.keepequal
- Drop targets for which the concatenatedsource_labels
do not matchtarget_label
.dropequal
- Drop targets for which the concatenatedsource_labels
do matchtarget_label
.lowercase
- Setstarget_label
to the lowercase form of the concatenatedsource_labels
.uppercase
- Setstarget_label
to the uppercase form of the concatenatedsource_labels
.
Finally, note that the regex capture groups can be referred to using either the
$CAPTURE_GROUP_NUMBER
or ${CAPTURE_GROUP_NUMBER}
notation.
selector block
The selector
block describes a Kubernetes label selector for ServiceMonitors.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
match_labels | map(string) | Label keys and values used to discover resources. | {} | no |
When the match_labels
argument is empty, all ServiceMonitor resources will be matched.
match_expression block
The match_expression
block describes a Kubernetes label matcher expression for
ServiceMonitors discovery.
The following arguments are supported:
Name | Type | Description | Default | Required |
---|---|---|---|---|
key | string | The label name to match against. | yes | |
operator | string | The operator to use when matching. | yes | |
values | list(string) | The values used when matching. | no |
The operator
argument must be one of the following strings:
"In"
"NotIn"
"Exists"
"DoesNotExist"
If there are multiple match_expressions
blocks inside of a selector
block, they are combined together with AND clauses.
clustering (beta)
Name | Type | Description | Default | Required |
---|---|---|---|---|
enabled | bool | Enables sharing targets with other cluster nodes. | false | yes |
When the agent is using using clustering, and enabled
is set to true,
then this component instance opts-in to participating in
the cluster to distribute scrape load between all cluster nodes.
Clustering assumes that all cluster nodes are running with the same
configuration file, and that all
prometheus.operator.servicemonitors
components that have opted-in to using clustering, over
the course of a scrape interval have the same configuration.
All prometheus.operator.servicemonitors
components instances opting in to clustering use target
labels and a consistent hashing algorithm to determine ownership for each of
the targets between the cluster peers. Then, each peer only scrapes the subset
of targets that it is responsible for, so that the scrape load is distributed.
When a node joins or leaves the cluster, every peer recalculates ownership and
continues scraping with the new target set. This performs better than hashmod
sharding where all nodes have to be re-distributed, as only 1/N of the
target’s ownership is transferred, but is eventually consistent (rather than
fully consistent like hashmod sharding is).
If the agent is not running in clustered mode, then the block is a no-op, and
prometheus.operator.servicemonitors
scrapes every target it receives in its arguments.
Exported fields
prometheus.operator.servicemonitors
does not export any fields. It forwards all metrics it scrapes to the receiver configures with the forward_to
argument.
Component health
prometheus.operator.servicemonitors
is reported as unhealthy when given an invalid configuration, Prometheus components fail to initialize, or the connection to the Kubernetes API could not be established properly.
Debug information
prometheus.operator.servicemonitors
reports the status of the last scrape for each configured
scrape job on the component’s debug endpoint, including discovered labels, and the last scrape time.
It also exposes some debug information for each ServiceMonitor it has discovered, including any errors found while reconciling the scrape configuration from the ServiceMonitor.
Debug metrics
Example
This example discovers all ServiceMonitors in your cluster, and forwards collected metrics to a
prometheus.remote_write
component.
prometheus.remote_write "staging" {
// Send metrics to a locally running Mimir.
endpoint {
url = "http://mimir:9009/api/v1/push"
basic_auth {
username = "example-user"
password = "example-password"
}
}
}
prometheus.operator.servicemonitors "services" {
forward_to = [prometheus.remote_write.staging.receiver]
}
This example will limit discovered ServiceMonitors to ones with the label team=ops
in a specific namespace: my-app
.
prometheus.operator.servicemonitors "services" {
forward_to = [prometheus.remote_write.staging.receiver]
namespaces = ["my-app"]
selector {
match_expression {
key = "team"
operator = "In"
values = ["ops"]
}
}
}
This example will apply additional relabel rules to discovered targets to filter by hostname. This may be useful if running the agent as a DaemonSet.
prometheus.operator.servicemonitors "services" {
forward_to = [prometheus.remote_write.staging.receiver]
rule {
action = "keep"
regex = env("HOSTNAME")
source_labels = ["__meta_kubernetes_pod_node_name"]
}
}