---
title: "mimir.alerts.kubernetes | Grafana Alloy documentation"
description: "Learn about mimir.alerts.kubernetes"
---

# `mimir.alerts.kubernetes`

> **EXPERIMENTAL**: This is an [experimental](/docs/release-life-cycle/) component. Experimental components are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental component, you must set the `stability.level` [flag](/docs/alloy/latest/reference/cli/run/) to `experimental`.

`mimir.alerts.kubernetes` discovers `AlertmanagerConfig` Kubernetes resources and loads them into a Mimir instance.

- You can specify multiple `mimir.alerts.kubernetes` components by giving them different labels.
- You can use [Kubernetes label selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors) to limit the `Namespace` and `AlertmanagerConfig` resources considered during reconciliation.
- Compatible with the Alertmanager APIs of Grafana Mimir, Grafana Cloud, and Grafana Enterprise Metrics.
- Compatible with the `AlertmanagerConfig` CRD from the [`prometheus-operator`](https://prometheus-operator.dev/).
- This component accesses the Kubernetes REST API from [within a Pod](https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/).

> Note
> 
> This component requires [Role-based access control (RBAC)](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to be set up in Kubernetes in order for Alloy to access it via the Kubernetes REST API.

`mimir.alerts.kubernetes` doesn’t support [clustering](../../../../get-started/clustering/).

## Usage

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
mimir.alerts.kubernetes "<LABEL>" {
  address       = "<MIMIR_URL>"
  global_config = "..."
}
```

## Arguments

You can use the following arguments with `mimir.alerts.kubernetes`:

Expand table

| Name                     | Type                | Description                                                                                                                        | Default | Required |
|--------------------------|---------------------|------------------------------------------------------------------------------------------------------------------------------------|---------|----------|
| `address`                | `string`            | URL of the Mimir Alertmanager.                                                                                                     |         | yes      |
| `global_config`          | `secret`            | [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) to be merged with AlertmanagerConfig CRDs. |         | yes      |
| `bearer_token_file`      | `string`            | File containing a bearer token to authenticate with.                                                                               |         | no       |
| `bearer_token`           | `secret`            | Bearer token to authenticate with.                                                                                                 |         | no       |
| `enable_http2`           | `bool`              | Whether HTTP2 is supported for requests.                                                                                           | `true`  | no       |
| `follow_redirects`       | `bool`              | Whether redirects returned by the server should be followed.                                                                       | `true`  | no       |
| `http_headers`           | `map(list(secret))` | Custom HTTP headers to be sent along with each request. The map key is the header name.                                            |         | no       |
| `no_proxy`               | `string`            | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.                                   |         | no       |
| `proxy_connect_header`   | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests.                                                                      |         | no       |
| `proxy_from_environment` | `bool`              | Use the proxy URL indicated by environment variables.                                                                              | `false` | no       |
| `proxy_url`              | `string`            | HTTP proxy to send requests through.                                                                                               |         | no       |
| `sync_interval`          | `duration`          | Amount of time between reconciliations with Mimir.                                                                                 | `"5m"`  | no       |
| `template_files`         | `map(string)`       | A map of Alertmanager [template files](/docs/mimir/latest/references/http-api/#set-alertmanager-configuration).                    | `{}`    | no       |

At most, one of the following can be provided:

- \[`authorization`]\[authorization] block
- \[`basic_auth`]\[basic\_auth] block
- [`bearer_token_file`](#arguments)argument
- [`bearer_token`](#arguments) argument
- \[`oauth2`]\[oauth2] block

`no_proxy` can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. `proxy_url` must be configured if `no_proxy` is configured.

`proxy_from_environment` uses the environment variables HTTP\_PROXY, HTTPS\_PROXY, and NO\_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO\_PROXY. `proxy_url` and `no_proxy` must not be configured if `proxy_from_environment` is configured.

`proxy_connect_header` should only be configured if `proxy_url` or `proxy_from_environment` are configured.

## Blocks

The following blocks are supported inside the definition of `mimir.alerts.kubernetes`:

No valid configuration blocks found.

### `authorization`

Expand table

| Name               | Type     | Description                                | Default | Required |
|--------------------|----------|--------------------------------------------|---------|----------|
| `credentials_file` | `string` | File containing the secret value.          |         | no       |
| `credentials`      | `secret` | Secret value.                              |         | no       |
| `type`             | `string` | Authorization type, for example, “Bearer”. |         | no       |

`credential` and `credentials_file` are mutually exclusive, and only one can be provided inside an `authorization` block.

> Warning
> 
> Using `credentials_file` causes the file to be read on every outgoing request. Use the `local.file` component with the `credentials` attribute instead to avoid unnecessary reads.

### `basic_auth`

Expand table

| Name            | Type     | Description                              | Default | Required |
|-----------------|----------|------------------------------------------|---------|----------|
| `password_file` | `string` | File containing the basic auth password. |         | no       |
| `password`      | `secret` | Basic auth password.                     |         | no       |
| `username`      | `string` | Basic auth username.                     |         | no       |

`password` and `password_file` are mutually exclusive, and only one can be provided inside a `basic_auth` block.

> Warning
> 
> Using `password_file` causes the file to be read on every outgoing request. Use the `local.file` component with the `password` attribute instead to avoid unnecessary reads.

### `alertmanagerconfig_selector` and `alertmanagerconfig_namespace_selector`

The `alertmanagerconfig_selector` and `alertmanagerconfig_namespace_selector` blocks describe a Kubernetes label selector for AlertmanagerConfig CRDs or namespace discovery.

The following arguments are supported:

Expand table

| Name           | Type          | Description                                       | Default | Required |
|----------------|---------------|---------------------------------------------------|---------|----------|
| `match_labels` | `map(string)` | Label keys and values used to discover resources. | `{}`    | yes      |

When the `match_labels` argument is empty, all resources are matched.

### `match_expression`

The `match_expression` block describes a Kubernetes label match expression for AlertmanagerConfig CRDs or namespace discovery.

The following arguments are supported:

Expand table

| Name       | Type           | Description                        | Default | Required |
|------------|----------------|------------------------------------|---------|----------|
| `key`      | `string`       | The label name to match against.   |         | yes      |
| `operator` | `string`       | The operator to use when matching. |         | yes      |
| `values`   | `list(string)` | The values used when matching.     |         | no       |

The `operator` argument should be one of the following strings:

- `"In"`
- `"NotIn"`
- `"Exists"`
- `"DoesNotExist"`

The `values` argument must not be provided when `operator` is set to `"Exists"` or `"DoesNotExist"`.

### `alertmanagerconfig_matcher`

The `alertmanagerconfig_matcher` block describes the strategy used by AlertmanagerConfig objects to match alerts in the routes and inhibition rules.

Depending on how this block is configured, the final Alertmanager config will have different [matchers](https://prometheus.io/docs/alerting/latest/configuration/#matcher) in its [route](https://prometheus.io/docs/alerting/latest/configuration/#route) section.

The following arguments are supported:

Expand table

| Name                     | Type     | Description                                                                                                          | Default         | Required                                                              |
|--------------------------|----------|----------------------------------------------------------------------------------------------------------------------|-----------------|-----------------------------------------------------------------------|
| `strategy`               | `string` | Strategy for adding matchers to AlertmanagerConfig CRDs.                                                             | `"OnNamespace"` | no                                                                    |
| `alertmanager_namespace` | `string` | Namespace to use when `alertmanagerconfig_matcher_strategy` is set to `"OnNamespaceExceptForAlertmanagerNamespace"`. |                 | only when `strategy` is `"OnNamespaceExceptForAlertmanagerNamespace"` |

The `strategy` argument should be one of the following strings:

- `"OnNamespace"`: Each AlertmanagerConfig object only matches alerts that have the `namespace` label set to the same namespace as the AlertmanagerConfig object.
- `"OnNamespaceExceptForAlertmanagerNamespace"`: The same as `"OnNamespace"`, except for AlertmanagerConfigs in the namespace given by `alertmanager_namespace`, which apply to all alerts.
- `"None"`: Every AlertmanagerConfig object applies to all alerts

`strategy` is similar to the [AlertmanagerConfigMatcherStrategy](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.AlertmanagerConfigMatcherStrategy) in Prometheus Operator, but it is configured in Alloy instead of in an Alertmanager CRD. Alloy doesn’t require an Alertmanager CRD.

### `oauth2`

Expand table

| Name                     | Type                | Description                                                                                      | Default | Required |
|--------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------|----------|
| `client_id`              | `string`            | OAuth2 client ID.                                                                                |         | no       |
| `client_secret_file`     | `string`            | File containing the OAuth2 client secret.                                                        |         | no       |
| `client_secret`          | `secret`            | OAuth2 client secret.                                                                            |         | no       |
| `endpoint_params`        | `map(string)`       | Optional parameters to append to the token URL.                                                  |         | no       |
| `no_proxy`               | `string`            | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. |         | no       |
| `proxy_connect_header`   | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests.                                    |         | no       |
| `proxy_from_environment` | `bool`              | Use the proxy URL indicated by environment variables.                                            | `false` | no       |
| `proxy_url`              | `string`            | HTTP proxy to send requests through.                                                             |         | no       |
| `scopes`                 | `list(string)`      | List of scopes to authenticate with.                                                             |         | no       |
| `token_url`              | `string`            | URL to fetch the token from.                                                                     |         | no       |

`client_secret` and `client_secret_file` are mutually exclusive, and only one can be provided inside an `oauth2` block.

> Warning
> 
> Using `client_secret_file` causes the file to be read on every outgoing request. Use the `local.file` component with the `client_secret` attribute instead to avoid unnecessary reads.

The `oauth2` block may also contain a separate `tls_config` sub-block.

`no_proxy` can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. `proxy_url` must be configured if `no_proxy` is configured.

`proxy_from_environment` uses the environment variables HTTP\_PROXY, HTTPS\_PROXY, and NO\_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO\_PROXY. `proxy_url` and `no_proxy` must not be configured if `proxy_from_environment` is configured.

`proxy_connect_header` should only be configured if `proxy_url` or `proxy_from_environment` are configured.

### `tls_config`

Expand table

| Name                   | Type     | Description                                              | Default | Required |
|------------------------|----------|----------------------------------------------------------|---------|----------|
| `ca_pem`               | `string` | CA PEM-encoded text to validate the server with.         |         | no       |
| `ca_file`              | `string` | CA certificate to validate the server with.              |         | no       |
| `cert_pem`             | `string` | Certificate PEM-encoded text for client authentication.  |         | no       |
| `cert_file`            | `string` | Certificate file for client authentication.              |         | no       |
| `insecure_skip_verify` | `bool`   | Disables validation of the server certificate.           |         | no       |
| `key_file`             | `string` | Key file for client authentication.                      |         | no       |
| `key_pem`              | `secret` | Key PEM-encoded text for client authentication.          |         | no       |
| `min_version`          | `string` | Minimum acceptable TLS version.                          |         | no       |
| `server_name`          | `string` | ServerName extension to indicate the name of the server. |         | no       |

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

- `ca_pem` and `ca_file`
- `cert_pem` and `cert_file`
- `key_pem` and `key_file`

When configuring client authentication, both the client certificate (using `cert_pem` or `cert_file`) and the client key (using `key_pem` or `key_file`) must be provided.

When `min_version` isn’t provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If `min_version` is provided, it must be set to one of the following strings:

- `"TLS10"` (TLS 1.0)
- `"TLS11"` (TLS 1.1)
- `"TLS12"` (TLS 1.2)
- `"TLS13"` (TLS 1.3)

## Exported fields

`mimir.alerts.kubernetes` doesn’t export any fields.

## Component health

`mimir.alerts.kubernetes` is reported as unhealthy if given an invalid configuration or an error occurs during reconciliation.

## Debug information

`mimir.alerts.kubernetes` doesn’t expose debug information.

## Debug metrics

Expand table

| Metric Name                                          | Type        | Description                                                              |
|------------------------------------------------------|-------------|--------------------------------------------------------------------------|
| `mimir_alerts_mimir_client_request_duration_seconds` | `histogram` | Duration of requests to the Mimir API.                                   |
| `mimir_alerts_config_updates_total`                  | `counter`   | Number of times the configuration has been updated.                      |
| `mimir_alerts_events_failed_total`                   | `counter`   | Number of events that failed to be processed, partitioned by event type. |
| `mimir_alerts_events_retried_total`                  | `counter`   | Number of events that were retried, partitioned by event type.           |
| `mimir_alerts_events_total`                          | `counter`   | Number of events processed, partitioned by event type.                   |

## Example

This example creates a `mimir.alerts.kubernetes` component which only loads namespace and `AlertmanagerConfig` resources if they contain an `alloy` label set to `yes`.

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
remote.kubernetes.configmap "default" {
  namespace = "default"
  name      = "alertmgr-global"
}

mimir.alerts.kubernetes "default" {
  address       = "http://mimir-nginx.mimir-test.svc:80"
  global_config = remote.kubernetes.configmap.default.data["glbl"]

  template_files = {
    `default_template` = 
`{{ define "__alertmanager" }}AlertManager{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}`,
  }

  alertmanagerconfig_selector {
      match_labels = {
          alloy = "yes",
      }
  }

  alertmanagerconfig_namespace_selector {
      match_labels = {
          alloy = "yes",
      }
  }
}
```

The following example is an RBAC configuration for Kubernetes. It authorizes Alloy to query the Kubernetes REST API:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: alloy
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: alloy
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["alertmanagerconfigs"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: alloy
subjects:
- kind: ServiceAccount
  name: alloy
  namespace: default
roleRef:
  kind: ClusterRole
  name: alloy
  apiGroup: rbac.authorization.k8s.io
```

The following is an example of a complete Kubernetes configuration:

Example Kubernetes configuration.

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: testing
  labels:
    alloy: "yes"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana-alloy
  namespace: testing
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-alloy
rules:
- apiGroups: [""]
  resources: ["namespaces", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["alertmanagerconfigs"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: grafana-alloy
subjects:
- kind: ServiceAccount
  name: grafana-alloy
  namespace: testing
roleRef:
  kind: ClusterRole
  name: grafana-alloy
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-alloy
spec:
  type: NodePort
  selector:
    app: grafana-alloy
  ports:
      # By default and for convenience, the `targetPort` is set to the same value as the `port` field.
    - port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: testing
  name: grafana-alloy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana-alloy
  template:
    metadata:
      labels:
        app: grafana-alloy
    spec:
      serviceAccount: grafana-alloy
      containers:
      - name: alloy
        image: grafana/alloy:latest
        imagePullPolicy: Never
        args:
        - run
        - /etc/config/config.alloy
        - --stability.level=experimental
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
        - name: config-volume
          configMap:
            name: alloy-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alloy-config
  namespace: testing
data:
  config.alloy: |
      remote.kubernetes.configmap "default" {
        namespace = "testing"
        name = "alertmgr-global"
      }

      mimir.alerts.kubernetes "default" {
        address = "http://mimir-nginx.mimir-test.svc:80"
        global_config = remote.kubernetes.configmap.default.data["glbl"]
        template_files = {
          `default_template` = 
      `{{ define "__alertmanager" }}AlertManager{{ end }}
      {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}`,
        }
        alertmanagerconfig_namespace_selector {
            match_labels = {
                alloy = "yes",
            }
        }
        alertmanagerconfig_selector {
            match_labels = {
                alloy = "yes",
            }
        }
      }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmgr-global
  namespace: testing
data:
  glbl: |
    global:
      resolve_timeout: 5m
      http_config:
        follow_redirects: true
        enable_http2: true
      smtp_hello: localhost
      smtp_require_tls: true
    route:
      receiver: "null"
    receivers:
    - name: "null"
    - name: "alloy-namespace/global-config/myreceiver"
    templates:
    - 'default_template'
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmgr-config1
  namespace: testing
  labels:
    alloy: "yes"
spec:
  route:
    receiver: "null"
    routes:
    - receiver: myamc
      continue: true
  receivers:
  - name: "null"
  - name: myamc
    webhookConfigs:
    - url: http://test.url
      httpConfig:
        followRedirects: true
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alertmgr-config2
  namespace: testing
  labels:
    alloy: "yes"
spec:
  route:
    receiver: "null"
    routes:
    - receiver: 'database-pager'
      groupWait: 10s
      matchers:
      - name: service
        value: webapp
  receivers:
  - name: "null"
  - name: "database-pager"
```

The Kubernetes configuration above creates the Alertmanager configuration below and sends it to Mimir:

Example merged configuration sent to Mimir.

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
template_files:
    default_template: |-
        {{ define "__alertmanager" }}AlertManager{{ end }}
        {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}
alertmanager_config: |
    global:
      resolve_timeout: 5m
      http_config:
        follow_redirects: true
        enable_http2: true
      smtp_hello: localhost
      smtp_require_tls: true
    route:
      receiver: "null"
      continue: false
      routes:
      - receiver: testing/alertmgr-config1/null
        matchers:
        - namespace="testing"
        continue: true
        routes:
        - receiver: testing/alertmgr-config1/myamc
          continue: true
      - receiver: testing/alertmgr-config2/null
        matchers:
        - namespace="testing"
        continue: true
        routes:
        - receiver: testing/alertmgr-config2/database-pager
          matchers:
          - service="webapp"
          continue: false
          group_wait: 10s
    receivers:
    - name: "null"
    - name: alloy-namespace/global-config/myreceiver
    - name: testing/alertmgr-config1/null
    - name: testing/alertmgr-config1/myamc
      webhook_configs:
      - send_resolved: false
        http_config:
          follow_redirects: true
          enable_http2: true
        url: <secret>
        url_file: ""
        max_alerts: 0
        timeout: 0s
    - name: testing/alertmgr-config2/null
    - name: testing/alertmgr-config2/database-pager
    templates:
    - default_template
```

You can add the `alertmanagerconfig_matcher` block to your Alloy configuration to remove the namespace matchers:

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
alertmanagerconfig_matcher {
  strategy = "None"
}
```

This results in the following final configuration:

Example merged configuration sent to Mimir.

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
template_files:
    default_template: |-
        {{ define "__alertmanager" }}AlertManager{{ end }}
        {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}
alertmanager_config: |
    global:
      resolve_timeout: 5m
      http_config:
        follow_redirects: true
        enable_http2: true
      smtp_hello: localhost
      smtp_require_tls: true
    route:
      receiver: "null"
      continue: false
      routes:
      - receiver: testing/alertmgr-config1/null
        continue: true
        routes:
        - receiver: testing/alertmgr-config1/myamc
          continue: true
      - receiver: testing/alertmgr-config2/null
        continue: true
        routes:
        - receiver: testing/alertmgr-config2/database-pager
          matchers:
          - service="webapp"
          continue: false
          group_wait: 10s
    receivers:
    - name: "null"
    - name: alloy-namespace/global-config/myreceiver
    - name: testing/alertmgr-config1/null
    - name: testing/alertmgr-config1/myamc
      webhook_configs:
      - send_resolved: false
        http_config:
          follow_redirects: true
          enable_http2: true
        url: <secret>
        url_file: ""
        max_alerts: 0
        timeout: 0s
    - name: testing/alertmgr-config2/null
    - name: testing/alertmgr-config2/database-pager
    templates:
    - default_template
```