Static mode Kubernetes operator (Beta) on Grafana Labs

Install the Operator with Helm

Wed, 12 Mar 2025 11:45:31 +0100

Install the Operator with Helm

In this guide, you’ll learn how to deploy Grafana Agent Operator into your Kubernetes cluster using the grafana-agent-operator Helm chart. To learn how to deploy Agent Operator without using Helm, see Install Grafana Agent Operator.

Note: If you are shipping your data to Grafana Cloud, use Kubernetes Monitoring to set up Agent Operator. Kubernetes Monitoring provides a simplified approach and preconfigured dashboards and alerts.

Before you begin

To deploy Agent Operator with Helm, make sure that you have the following:

A Kubernetes cluster
The kubectl command-line client installed and configured on your machine
The helm command-line client installed and configured on your machine

Note: Agent Operator is currently in beta and its custom resources are subject to change.

Install the Agent Operator Helm Chart

In this section, you’ll install the grafana-agent-operator Helm chart into your Kubernetes cluster. This will install the latest version of Agent Operator and its Custom Resource Definitions (CRDs). The chart configures Operator to maintain a Service that lets you scrape kubelets using a ServiceMonitor.

To install the Agent Operator Helm chart:

Add and update the grafana Helm chart repo:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install the chart, replacing my-release with your release name:
Bash
```
helm install my-release grafana/grafana-agent-operator
```
If you want to modify the default parameters, you can create a values.yaml file and pass it to helm install:
Bash
```
helm install my-release grafana/grafana-agent-operator -f values.yaml
```
If you want to deploy Agent Operator into a namespace other than default, use the -n flag:
Bash
```
helm install my-release grafana/grafana-agent-operator -f values.yaml -n my-namespace
```
You can find a list of configurable template parameters in the Helm chart repository.
Once you’ve successfully deployed the Helm release, confirm that Agent Operator is up and running:
Bash
```
kubectl get pod
kubectl get svc
```
You should see an Agent Operator Pod in RUNNING state, and a kubelet service. Depending on your setup, this could take a moment.

Deploy the Grafana Agent Operator resources

Agent Operator is now up and running. Next, you need to install a Grafana Agent for Agent Operator to run for you. To do so, follow the instructions in the Deploy the Grafana Agent Operator resources topic. To learn more about the custom resources Agent Operator provides and their hierarchy, see Grafana Agent Operator architecture.

Install the Operator

Wed, 12 Mar 2025 11:45:31 +0100

Install the Operator

In this guide, you’ll learn how to deploy Grafana Agent Operator into your Kubernetes cluster. This guide does not use Helm. To learn how to deploy Agent Operator using the grafana-agent-operator Helm chart, see Install Grafana Agent Operator with Helm.

Note: If you are shipping your data to Grafana Cloud, use Kubernetes Monitoring to set up Agent Operator. Kubernetes Monitoring provides a simplified approach and preconfigured dashboards and alerts.

Before you begin

To deploy Agent Operator, make sure that you have the following:

A Kubernetes cluster
The kubectl command-line client installed and configured on your machine

Note: Agent Operator is currently in beta and its custom resources are subject to change.

Deploy the Agent Operator Custom Resource Definitions (CRDs)

Before you can create the custom resources for a Grafana Agent deployment, you need to deploy the Custom Resource Definitions to the cluster. These definitions describe the schema that the custom resources will conform to. This is also required for Grafana Agent Operator to run; it will fail if it can’t find the Custom Resource Definitions of objects it is looking to use. To learn more about the custom resources Agent Operator provides and their hierarchy, see Grafana Agent Operator architecture.

You can find the set of Custom Resource Definitions for Grafana Agent Operator in the Grafana Agent repository under operations/agent-static-operator/crds.

To deploy the CRDs:

Clone the agent repo and then apply the CRDs from the root of the agent repository:
```
kubectl apply -f production/operator/crds
```
This step must be completed before installing Agent Operator—it will fail to start if the CRDs do not exist.
To check that the CRDs are deployed to your Kubernetes cluster and to access documentation for each resource, use kubectl explain <resource>.

For example, kubectl explain GrafanaAgent describes the GrafanaAgent CRD, and kubectl explain GrafanaAgent.spec gives you information on its spec field.

Install Grafana Agent Operator

Next, install Agent Operator by applying the Agent Operator deployment schema.

To install Agent Operator:

Copy the following deployment schema to a file, updating the namespace if needed:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-agent-operator
  namespace: default
  labels:
    app: grafana-agent-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana-agent-operator
  template:
    metadata:
      labels:
        app: grafana-agent-operator
    spec:
      serviceAccountName: grafana-agent-operator
      containers:
      - name: operator
        image: grafana/agent-operator:v0.43.4
        args:
        - --kubelet-service=default/kubelet
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana-agent-operator
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-agent-operator
rules:
- apiGroups: [monitoring.grafana.com]
  resources:
  - grafanaagents
  - metricsinstances
  - logsinstances
  - podlogs
  - integrations
  verbs: [get, list, watch]
- apiGroups: [monitoring.coreos.com]
  resources:
  - podmonitors
  - probes
  - servicemonitors
  verbs: [get, list, watch]
- apiGroups: [""]
  resources:
  - namespaces
  - nodes
  verbs: [get, list, watch]
- apiGroups: [""]
  resources:
  - secrets
  - services
  - configmaps
  - endpoints
  verbs: [get, list, watch, create, update, patch, delete]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - daemonsets
  - deployments
  verbs: [get, list, watch, create, update, patch, delete]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: grafana-agent-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: grafana-agent-operator
subjects:
- kind: ServiceAccount
  name: grafana-agent-operator
  namespace: default

Roll out the deployment in your cluster using kubectl apply -f followed by your deployment filename.

Note: If you want to run Agent Operator locally, make sure your kubectl context is correct. Running locally uses your current kubectl context. If it is set to your production environment, you could accidentally deploy a new Grafana Agent to production. Install CRDs on the cluster prior to running locally. Afterwards, you can run Agent Operator using go run ./cmd/grafana-agent-operator.

Deploy the Grafana Agent Operator resources

Deploy Operator resources

Wed, 12 Mar 2025 11:45:31 +0100

Deploy Operator resources

To start collecting telemetry data, you need to roll out Grafana Agent Operator custom resources into your Kubernetes cluster. Before you can create the custom resources, you must first apply the Agent Custom Resource Definitions (CRDs) and install Agent Operator, with or without Helm. If you haven’t yet taken these steps, follow the instructions in one of the following topics:

Follow the steps in this guide to roll out the Grafana Agent Operator custom resources to:

Scrape and ship cAdvisor and kubelet metrics to a Prometheus-compatible metrics endpoint.
Collect and ship your Pods’ container logs to a Loki-compatible logs endpoint.

The hierarchy of custom resources is as follows:

GrafanaAgent
- MetricsInstance
  - PodMonitor
  - Probe
  - ServiceMonitor
- LogsInstance
  - PodLogs

To learn more about the custom resources Agent Operator provides and their hierarchy, see Grafana Agent Operator architecture.

Note
Agent Operator is currently in beta and its custom resources are subject to change.

Before you begin

Before you begin, make sure that you have deployed the Grafana Agent Operator CRDs and installed Agent Operator into your cluster. See Install Grafana Agent Operator with Helm or Install Grafana Agent Operator for instructions.

Deploy the GrafanaAgent resource

In this section, you’ll roll out a GrafanaAgent resource. See Grafana Agent Operator architecture for a discussion of the resources in the GrafanaAgent resource hierarchy.

Note
Due to the variety of possible deployment architectures, the official Agent Operator Helm chart does not provide built-in templates for the custom resources described in this guide. You must configure and deploy these manually as described in this section. We recommend templating and adding the following manifests to your own in-house Helm charts and GitOps flows.

To deploy the GrafanaAgent resource:

Copy the following manifests to a file:

apiVersion: monitoring.grafana.com/v1alpha1
kind: GrafanaAgent
metadata:
  name: grafana-agent
  namespace: default
  labels:
    app: grafana-agent
spec:
  image: grafana/agent:v0.43.4
  integrations:
    selector:
      matchLabels:
          agent: grafana-agent-integrations
  logLevel: info
  serviceAccountName: grafana-agent
  metrics:
    instanceSelector:
      matchLabels:
        agent: grafana-agent-metrics
    externalLabels:
      cluster: cloud

  logs:
    instanceSelector:
      matchLabels:
        agent: grafana-agent-logs

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana-agent
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-agent
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  - nodes/metrics
  - services
  - endpoints
  - pods
  - events
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  - /metrics/cadvisor
  verbs:
  - get

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: grafana-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: grafana-agent
subjects:
- kind: ServiceAccount
  name: grafana-agent
  namespace: default

In the first manifest, the GrafanaAgent resource:

Specifies an Agent image version.
Specifies MetricsInstance and LogsInstance selectors. These search for MetricsInstances and LogsInstances in the same namespace with labels matching agent: grafana-agent-metrics and agent: grafana-agent-logs, respectively.
Sets a cluster: cloud label for all metrics shipped to your Prometheus-compatible endpoint. Change this label to your cluster name. To search for MetricsInstances or LogsInstances in a different namespace, use the instanceNamespaceSelector field. To learn more about this field, see the GrafanaAgent CRD specification.

Customize the manifests as needed and roll them out to your cluster using kubectl apply -f followed by the filename.

This step creates a ServiceAccount, ClusterRole, and ClusterRoleBinding for the GrafanaAgent resource.

Deploying a GrafanaAgent resource on its own does not spin up Agent Pods. Agent Operator creates Agent Pods once MetricsInstance and LogsIntance resources have been created. Follow the instructions in the Deploy a MetricsInstance resource and Deploy LogsInstance and PodLogs resources sections to create these resources.

Disable feature flags reporting

To disable the reporting usage of feature flags to Grafana, set disableReporting field to true.

Disable support bundle generation

To disable the support bundles functionality, set the disableSupportBundle field to true.

Deploy a MetricsInstance resource

Next, you’ll roll out a MetricsInstance resource. MetricsInstance resources define a remote_write sink for metrics and configure one or more selectors to watch for creation and updates to *Monitor objects. These objects allow you to define Agent scrape targets via Kubernetes manifests:

To deploy a MetricsInstance resource:

Copy the following manifest to a file:

apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
  name: primary
  namespace: default
  labels:
    agent: grafana-agent-metrics
spec:
  remoteWrite:
  - url: your_remote_write_URL
    basicAuth:
      username:
        name: primary-credentials-metrics
        key: username
      password:
        name: primary-credentials-metrics
        key: password

  # Supply an empty namespace selector to look in all namespaces. Remove
  # this to only look in the same namespace as the MetricsInstance CR
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      instance: primary

  # Supply an empty namespace selector to look in all namespaces. Remove
  # this to only look in the same namespace as the MetricsInstance CR.
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      instance: primary

  # Supply an empty namespace selector to look in all namespaces. Remove
  # this to only look in the same namespace as the MetricsInstance CR.
  probeNamespaceSelector: {}
  probeSelector:
    matchLabels:
      instance: primary

Replace the remote_write URL and customize the namespace and label configuration as necessary.

This step associates the MetricsInstance resource with the agent: grafana-agent GrafanaAgent resource deployed in the previous step. The MetricsInstance resource watches for creation and updates to *Monitors with the instance: primary label.

Once you’ve rolled out the manifest, create the basicAuth credentials using a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: primary-credentials-metrics
  namespace: default
stringData:
  username: 'your_cloud_prometheus_username'
  password: 'your_cloud_prometheus_API_key'

If you’re using Grafana Cloud, you can find your hosted Loki endpoint username and password by clicking Details on the Loki tile on the Grafana Cloud Portal. If you want to base64-encode these values yourself, use data instead of stringData.

Once you’ve rolled out the MetricsInstance and its Secret, you can confirm that the MetricsInstance Agent is up and running using kubectl get pod. Since you haven’t defined any monitors yet, this Agent doesn’t have any scrape targets defined. In the next section, you’ll create scrape targets for the cAdvisor and kubelet endpoints exposed by the kubelet service in the cluster.

Create ServiceMonitors for kubelet and cAdvisor endpoints

Next, you’ll create ServiceMonitors for kubelet and cAdvisor metrics exposed by the kubelet service. Every Node in your cluster exposes kubelet and cAdvisor metrics at /metrics and /metrics/cadvisor, respectively. Agent Operator creates a kubelet service that exposes these Node endpoints so that they can be scraped using ServiceMonitors.

To scrape the kubelet and cAdvisor endpoints:

Copy the following kubelet ServiceMonitor manifest to a file, then roll it out in your cluster using kubectl apply -f followed by the filename.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    instance: primary
  name: kubelet-monitor
  namespace: default
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    interval: 60s
    metricRelabelings:
    - action: keep
      regex: kubelet_cgroup_manager_duration_seconds_count|go_goroutines|kubelet_pod_start_duration_seconds_count|kubelet_runtime_operations_total|kubelet_pleg_relist_duration_seconds_bucket|volume_manager_total_volumes|kubelet_volume_stats_capacity_bytes|container_cpu_usage_seconds_total|container_network_transmit_bytes_total|kubelet_runtime_operations_errors_total|container_network_receive_bytes_total|container_memory_swap|container_network_receive_packets_total|container_cpu_cfs_periods_total|container_cpu_cfs_throttled_periods_total|kubelet_running_pod_count|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate|container_memory_working_set_bytes|storage_operation_errors_total|kubelet_pleg_relist_duration_seconds_count|kubelet_running_pods|rest_client_request_duration_seconds_bucket|process_resident_memory_bytes|storage_operation_duration_seconds_count|kubelet_running_containers|kubelet_runtime_operations_duration_seconds_bucket|kubelet_node_config_error|kubelet_cgroup_manager_duration_seconds_bucket|kubelet_running_container_count|kubelet_volume_stats_available_bytes|kubelet_volume_stats_inodes|container_memory_rss|kubelet_pod_worker_duration_seconds_count|kubelet_node_name|kubelet_pleg_relist_interval_seconds_bucket|container_network_receive_packets_dropped_total|kubelet_pod_worker_duration_seconds_bucket|container_start_time_seconds|container_network_transmit_packets_dropped_total|process_cpu_seconds_total|storage_operation_duration_seconds_bucket|container_memory_cache|container_network_transmit_packets_total|kubelet_volume_stats_inodes_used|up|rest_client_requests_total
      sourceLabels:
      - __name__
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    - action: replace
      targetLabel: job
      replacement: integrations/kubernetes/kubelet
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet

Copy the following cAdvisor ServiceMonitor manifest to a file, then roll it out in your cluster using kubectl apply -f followed by the filename.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    instance: primary
  name: cadvisor-monitor
  namespace: default
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    honorTimestamps: false
    interval: 60s
    metricRelabelings:
    - action: keep
      regex: kubelet_cgroup_manager_duration_seconds_count|go_goroutines|kubelet_pod_start_duration_seconds_count|kubelet_runtime_operations_total|kubelet_pleg_relist_duration_seconds_bucket|volume_manager_total_volumes|kubelet_volume_stats_capacity_bytes|container_cpu_usage_seconds_total|container_network_transmit_bytes_total|kubelet_runtime_operations_errors_total|container_network_receive_bytes_total|container_memory_swap|container_network_receive_packets_total|container_cpu_cfs_periods_total|container_cpu_cfs_throttled_periods_total|kubelet_running_pod_count|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate|container_memory_working_set_bytes|storage_operation_errors_total|kubelet_pleg_relist_duration_seconds_count|kubelet_running_pods|rest_client_request_duration_seconds_bucket|process_resident_memory_bytes|storage_operation_duration_seconds_count|kubelet_running_containers|kubelet_runtime_operations_duration_seconds_bucket|kubelet_node_config_error|kubelet_cgroup_manager_duration_seconds_bucket|kubelet_running_container_count|kubelet_volume_stats_available_bytes|kubelet_volume_stats_inodes|container_memory_rss|kubelet_pod_worker_duration_seconds_count|kubelet_node_name|kubelet_pleg_relist_interval_seconds_bucket|container_network_receive_packets_dropped_total|kubelet_pod_worker_duration_seconds_bucket|container_start_time_seconds|container_network_transmit_packets_dropped_total|process_cpu_seconds_total|storage_operation_duration_seconds_bucket|container_memory_cache|container_network_transmit_packets_total|kubelet_volume_stats_inodes_used|up|rest_client_requests_total
      sourceLabels:
      - __name__
    path: /metrics/cadvisor
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    - action: replace
      targetLabel: job
      replacement: integrations/kubernetes/cadvisor
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet

These two ServiceMonitors configure Agent to scrape all the kubelet and cAdvisor endpoints in your Kubernetes cluster (one of each per Node). In addition, it defines a job label which you can update (it is preset here for compatibility with Grafana Cloud’s Kubernetes integration). It also provides an allowlist containing a core set of Kubernetes metrics to reduce remote metrics usage. If you don’t need this allowlist, you can omit it, however, your metrics usage will increase significantly.

When you’re done, Agent should now be shipping kubelet and cAdvisor metrics to your remote Prometheus endpoint. To check this in Grafana Cloud, go to your dashboards, select Integration - Kubernetes, then select Kubernetes / Kubelet.

Deploy LogsInstance and PodLogs resources

Next, you’ll deploy a LogsInstance resource to collect logs from your cluster Nodes and ship these to your remote Loki endpoint. Agent Operator deploys a DaemonSet of Agents in your cluster that will tail log files defined in PodLogs resources.

To deploy the LogsInstance resource into your cluster:

Copy the following manifest to a file, then roll it out in your cluster using kubectl apply -f followed by the filename.
YAML
```
apiVersion: monitoring.grafana.com/v1alpha1
kind: LogsInstance
metadata:
  name: primary
  namespace: default
  labels:
    agent: grafana-agent-logs
spec:
  clients:
  - url: your_remote_logs_URL
    basicAuth:
      username:
        name: primary-credentials-logs
        key: username
      password:
        name: primary-credentials-logs
        key: password

  # Supply an empty namespace selector to look in all namespaces. Remove
  # this to only look in the same namespace as the LogsInstance CR
  podLogsNamespaceSelector: {}
  podLogsSelector:
    matchLabels:
      instance: primary
```
This LogsInstance picks up PodLogs resources with the instance: primary label. Be sure to set the Loki URL to the correct push endpoint. For Grafana Cloud, this will look similar to logs-prod-us-central1.grafana.net/loki/api/v1/push, however check the Grafana Cloud Portal to confirm by clicking Details on the Loki tile.

Also note that this example uses the agent: grafana-agent-logs label, which associates this LogsInstance with the GrafanaAgent resource defined earlier. This means that it will inherit requests, limits, affinities and other properties defined in the GrafanaAgent custom resource.
To create the Secret for the LogsInstance resource, copy the following Secret manifest to a file, then roll it out in your cluster using kubectl apply -f followed by the filename.
YAML
```
apiVersion: v1
kind: Secret
metadata:
  name: primary-credentials-logs
  namespace: default
stringData:
  username: 'your_username_here'
  password: 'your_password_here'
```
If you’re using Grafana Cloud, you can find your hosted Loki endpoint username and password by clicking Details on the Loki tile on the Grafana Cloud Portal. If you want to base64-encode these values yourself, use data instead of stringData.
Copy the following PodLogs manifest to a file, then roll it to your cluster using kubectl apply -f followed by the filename. The manifest defines your logging targets. Agent Operator turns this into Agent configuration for the logs subsystem, and rolls it out to the DaemonSet of logging Agents.

Note
The following is a minimal working example which you should adapt to your production needs.
YAML
```
apiVersion: monitoring.grafana.com/v1alpha1
kind: PodLogs
metadata:
  labels:
    instance: primary
  name: kubernetes-pods
  namespace: default
spec:
  pipelineStages:
    - docker: {}
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels: {}
```
This example tails container logs for all Pods in the default namespace. You can restrict the set of matched Pods by using the matchLabels selector. You can also set additional pipelineStages and create relabelings to add or modify log line labels. To learn more about the PodLogs specification and available resource fields, see the PodLogs CRD.

The above PodLogs resource adds the following labels to log lines:
- namespace
- service
- pod
- container
- job (set to PodLogs_namespace/PodLogs_name)
- __path__ (the path to log files, set to /var/log/pods/*$1/*.log where $1 is __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name)
To learn more about this configuration format and other available labels, see the Promtail Scraping documentation. Agent Operator loads this configuration into the LogsInstance agents automatically.

The DaemonSet of logging agents should be tailing your container logs, applying default labels to the log lines, and shipping them to your remote Loki endpoint.

Summary

You’ve now rolled out the following into your cluster:

A GrafanaAgent resource that discovers one or more MetricsInstance and LogsInstances resources.
A MetricsInstance resource that defines where to ship collected metrics.
A ServiceMonitor resource to collect cAdvisor and kubelet metrics.
A LogsInstance resource that defines where to ship collected logs.
A PodLogs resource to collect container logs from Kubernetes Pods.

What’s next

You can verify that everything is working correctly by navigating to your Grafana instance and querying your Loki and Prometheus data sources.

Tip: You can deploy multiple GrafanaAgent resources to isolate allocated resources to the agent pods. By default, the GrafanaAgent resource determines the resources of all deployed agent containers. However, you might want different memory limits for metrics versus logs.

Architecture

Wed, 12 Mar 2025 11:45:31 +0100

Architecture

Grafana Agent Operator works by watching for Kubernetes custom resources that specify how to collect telemetry data from your Kubernetes cluster and where to send it. Agent Operator manages corresponding Grafana Agent deployments in your cluster by watching for changes against the custom resources.

Grafana Agent Operator works in two phases—it discovers a hierarchy of custom resources and it reconciles that hierarchy into a Grafana Agent deployment.

Custom resource hierarchy

The root of the custom resource hierarchy is the GrafanaAgent resource—the primary resource Agent Operator looks for. GrafanaAgent is called the root because it discovers other sub-resources, MetricsInstance and LogsInstance. The GrafanaAgent resource endows them with Pod attributes defined in the GrafanaAgent specification, for example, Pod requests, limits, affinities, and tolerations, and defines the Grafana Agent image. You can only define Pod attributes at the GrafanaAgent level. They are propagated to MetricsInstance and LogsInstance Pods.

The full hierarchy of custom resources is as follows:

GrafanaAgent
- MetricsInstance
  - PodMonitor
  - Probe
  - ServiceMonitor
- LogsInstance
  - PodLogs

The following table describes these custom resources:

Custom resource	description
`GrafanaAgent`	Discovers one or more `MetricsInstance` and `LogsInstance` resources.
`MetricsInstance`	Defines where to ship collected metrics. This rolls out a Grafana Agent StatefulSet that will scrape and ship metrics to a `remote_write` endpoint.
`ServiceMonitor`	Collects cAdvisor and kubelet metrics. This configures the `MetricsInstance` / Agent StatefulSet
`LogsInstance`	Defines where to ship collected logs. This rolls out a Grafana Agent DaemonSet that will tail log files on your cluster nodes.
`PodLogs`	Collects container logs from Kubernetes Pods. This configures the `LogsInstance` / Agent DaemonSet.

Most of the Grafana Agent Operator resources have the ability to reference a ConfigMap or a Secret. All referenced ConfigMaps or Secrets are added into the resource hierarchy.

When a hierarchy is established, each item is watched for changes. Any changed item causes a reconcile of the root GrafanaAgent resource, either creating, modifying, or deleting the corresponding Grafana Agent deployment.

A single resource can belong to multiple hierarchies. For example, if two GrafanaAgents use the same Probe, modifying that Probe causes both GrafanaAgents to be reconciled.

To set up monitoring, Grafana Agent Operator works in the following two phases:

Builds (discovers) a hierarchy of custom resources.
Reconciles that hierarchy into a Grafana Agent deployment.

Agent Operator also performs sharding and replication and adds labels to every metric.

How Agent Operator builds the custom resource hierarchy

Grafana Agent Operator builds the hierarchy using label matching on the custom resources. The following figure illustrates the matching. The GrafanaAgent picks up the MetricsInstance and LogsInstance that match the label instance: primary. The instances pick up the resources the same way.

To validate the Secrets

The generated configurations are saved in Secrets. To download and validate them manually, use the following commands:

$ kubectl get secrets <???>-logs-config -o json | jq -r '.data."agent.yml"' | base64 --decode
$ kubectl get secrets <???>-config -o json | jq -r '.data."agent.yml"' | base64 --decode

How Agent Operator reconciles the custom resource hierarchy

When a resource hierarchy is created, updated, or deleted, a reconcile occurs. When a GrafanaAgent resource is deleted, the corresponding Grafana Agent deployment will also be deleted.

Reconciling creates the following cluster resources:

A Secret that holds the Grafana Agent configuration is generated.
A Secret that holds all referenced Secrets or ConfigMaps from the resource hierarchy is generated. This ensures that Secrets referenced from a custom resource in another namespace can still be read.
A Service is created to govern the StatefulSets that are generated.
One StatefulSet per Prometheus shard is created.

PodMonitors, Probes, and ServiceMonitors are turned into individual scrape jobs which all use Kubernetes Service Discovery (SD).

Sharding and replication

The GrafanaAgent resource can specify a number of shards. Each shard results in the creation of a StatefulSet with a hashmod + keep relabel_config per job:

- source_labels: [__address__]
  target_label: __tmp_hash
  modulus: NUM_SHARDS
  action: hashmod
- source_labels: [__tmp_hash]
  regex: CURRENT_STATEFULSET_SHARD
  action: keep

This allows for horizontal scaling capabilities, where each shard will handle roughly 1/N of the total scrape load. Note that this does not use consistent hashing, which means changing the number of shards will cause anywhere between 1/N to N targets to reshuffle.

The sharding mechanism is borrowed from the Prometheus Operator.

The number of replicas can be defined, similarly to the number of shards. This creates deduplicate shards. This must be paired with a remote_write system that can perform HA deduplication. Grafana Cloud and Mimir provide this out of the box, and the Grafana Agent Operator defaults support these two systems.

The total number of created metrics pods will be the product of numShards * numReplicas.

Added labels

Two labels are added by default to every metric:

cluster, representing the GrafanaAgent deployment. Holds the value of <GrafanaAgent.metadata.namespace>/<GrafanaAgent.metadata.name>.
__replica__, representing the replica number of the Agent. This label works out of the box with Grafana Cloud and Cortex’s HA deduplication.

The shard number is not added as a label, as sharding is designed to be transparent on the receiver end.

Enable sharding and replication

To enable sharding and replication, you must set the shards and replicas properties in the Grafana Agent configuration file. For example, the following configuration file would shard the data into three shards and replicate each shard to two other Grafana Agent instances:

shards: 3
replicas: 2

You can also enable sharding and replication by setting the shards and replicas arguments when you start the Grafana Agent.

Examples

The following examples show you how to enable sharding and replication in a Kubernetes environment.

To shard the data into three shards and replicate each shard to two other Grafana Agent instances, you would use the following deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-agent
spec:
  replicas: 3
  selector:
    matchLabels:
      app: grafana-agent
  template:
    metadata:
      labels:
        app: grafana-agent
    spec:
      containers:
      - name: grafana-agent
        image: grafana/agent:latest
        args:
        - "--shards=3"
        - "--replicas=2"

To shard the data into 10 shards and replicate each shard to three other Grafana Agent instances, you would use the following deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-agent
spec:
  replicas: 10
  selector:
    matchLabels:
      app: grafana-agent
  template:
    metadata:
      labels:
        app: grafana-agent
    spec:
      containers:
      - name: grafana-agent
        image: grafana/agent:latest
        args:
        - "--shards=10"
        - "--replicas=3"

Set up integrations

Wed, 12 Mar 2025 11:45:31 +0100

Set up integrations

This topic provides examples of setting up Grafana Agent Operator integrations, including node_exporter and mysqld_exporter.

Before you begin

Also, make sure that you deploy the GrafanaAgent resource and the yaml you use has the integrations definition under spec.

Important: The field name under the spec section of the manifest must contain the name of the integration to be installed according to the list of integrations defined here.

Important: The value of the metrics_instance field needs to be in the format <namespace>/<name>, with namespace and name matching the values defined in the metadata section from the MetricsInstance resource as explained in deploy a MetricsInstance resource

Set up an Agent Operator node_exporter integration

The Agent Operator node_exporter integration lets you monitor your hardware and OS metrics from Unix-based machines, including Linux machines.

To set up a node_exporter integration:

Copy the following manifest to a file:

apiVersion: monitoring.grafana.com/v1alpha1
kind: Integration
metadata:
 name: node-exporter
 namespace: default
 labels:
   agent: grafana-agent-integrations
spec:
 name: node_exporter
 type:
   allNodes: true
   unique: true
 config:
   autoscrape:
     enable: true
     metrics_instance: default/primary
   rootfs_path: /default/node_exporter/rootfs
   sysfs_path: /default/node_exporter/sys
   procfs_path: /default/node_exporter/proc
 volumeMounts:
   - mountPath: /default/node_exporter/proc
     name: proc
   - mountPath: /default/node_exporter/sys
     name: sys
   - mountPath: /default/node_exporter/rootfs
     name: root
 volumes:
   - name: proc
     hostPath:
       path: /proc
   - name: sys
     hostPath:
       path: /sys
   - name: root
     hostPath:
       path: /root

Customize the manifest as needed and roll it out to your cluster using kubectl apply -f followed by the filename.

The manifest causes Agent Operator to create an instance of a grafana-agent-integrations-deploy resource that exports Node metrics.

Set up an Agent Operator mysqld_exporter integration

The Agent Operator mysqld_exporter integration is an embedded version of mysqld_exporter that lets you collect metrics from MySQL servers.

To set up a mysqld_exporter integration:

Copy the following manifest to a file:

apiVersion: monitoring.grafana.com/v1alpha1
kind: Integration
metadata:
 name: mysqld-exporter
 namespace: default
 labels:
   agent: grafana-agent-integrations
spec:
 name: mysql
 type:
   allNodes: true
   unique: true
 config:
   autoscrape:
     enable: true
     metrics_instance: default/primary
   data_source_name: root@(server-a:3306)/

Customize the manifest as needed and roll it out to your cluster using kubectl apply -f followed by the filename.

The manifest causes Agent Operator to create an instance of a grafana-agent-integrations-deploy resource that exports MySQL metrics.

Add custom scrape jobs

Wed, 11 Sep 2024 17:43:35 +0000

Add custom scrape jobs

Sometimes you want to add a scrape job for something that isn’t supported by the standard set of Prometheus Operator CRDs. A common example of this is node-level metrics.

To do this, you’ll need to write custom scrape configs and store it in a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: extra-jobs
  namespace: operator
stringData:
  jobs.yaml: |
    <SCRAPE CONFIGS>

Replace <SCRAPE CONFIGS> above with the array of Prometheus scrape jobs to include.

For example, to collect metrics from Kubelet and cAdvisor, use the following:

apiVersion: v1
kind: Secret
metadata:
  name: extra-jobs
  namespace: operator
stringData:
  jobs.yaml: |
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: integrations/kubernetes/kubelet
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        source_labels: [__meta_kubernetes_node_name]
        replacement: /api/v1/nodes/$1/proxy/metrics
        target_label: __metrics_path__
      - action: hashmod
        modulus: $(SHARDS)
        source_labels:
        - __address__
        target_label: __tmp_hash
      - action: keep
        regex: $(SHARD)
        source_labels:
        - __tmp_hash
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: integrations/kubernetes/cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      - action: hashmod
        modulus: $(SHARDS)
        source_labels:
        - __address__
        target_label: __tmp_hash
      - action: keep
        regex: $(SHARD)
        source_labels:
        - __tmp_hash
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Note that you should always add these two relabel_configs for each custom job:

- action: hashmod
  modulus: $(SHARDS)
  source_labels:
  - __address__
  target_label: __tmp_hash
- action: keep
  regex: $(SHARD)
  source_labels:
  - __tmp_hash

These rules ensure if your GrafanaAgent has multiple metrics shards, only one pod per replica will collect metrics for each job.

Once your Secret is defined, you’ll then need to add a additionalScrapeConfigs field to your MetricsInstance:

apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
  labels:
    name: grafana-agent
  name: primary
  namespace: operator
spec:
  additionalScrapeConfigs:
    name: extra-jobs
    key: jobs.yaml
  # ... Other settings ...

The Secret MUST be in the same namespace as the MetricsInstance.

Custom Resource Definition Reference

Wed, 11 Sep 2024 17:43:35 +0000

Custom Resource Definition Reference

Resource Types:

Deployment

Deployment is a set of discovered resources relative to a GrafanaAgent. The tree of resources contained in a Deployment form the resource hierarchy used for reconciling a GrafanaAgent.

Fields

Field	Description
apiVersion	string `monitoring.grafana.com/v1alpha1`
kind	string `Deployment`
`Agent` GrafanaAgent	Root resource in the deployment.
`Metrics` []MetricsDeployment	Metrics resources discovered by Agent.
`Logs` []LogsDeployment	Logs resources discovered by Agent.
`Integrations` []IntegrationsDeployment	Integrations resources discovered by Agent.
`Secrets` github.com/grafana/agent/static/operator/assets.SecretStore	The full list of Secrets referenced by resources in the Deployment.

GrafanaAgent

(Appears on:Deployment) GrafanaAgent defines a Grafana Agent deployment.

Fields

Field	Description
apiVersion	string `monitoring.grafana.com/v1alpha1`
kind	string `GrafanaAgent`
`metadata` Kubernetes meta/v1.ObjectMeta	Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` GrafanaAgentSpec	Spec holds the specification of the desired behavior for the Grafana Agent cluster.
`logLevel` string	LogLevel controls the log level of the generated pods. Defaults to "info" if not set.
`logFormat` string	LogFormat controls the logging format of the generated pods. Defaults to "logfmt" if not set.
`apiServer` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.APIServerConfig	APIServerConfig lets you specify a host and auth methods to access the Kubernetes API server. If left empty, the Agent assumes that it is running inside of the cluster and will discover API servers automatically and use the pod's CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount.
`podMetadata` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.EmbeddedObjectMetadata	PodMetadata configures Labels and Annotations which are propagated to created Grafana Agent pods.
`version` string	Version of Grafana Agent to be deployed.
`paused` bool	Paused prevents actions except for deletion to be performed on the underlying managed objects.
`image` string	Image, when specified, overrides the image used to run Agent. Specify the image along with a tag. You still need to set the version to ensure Grafana Agent Operator knows which version of Grafana Agent is being configured.
`configReloaderVersion` string	Version of Config Reloader to be deployed.
`configReloaderImage` string	Image, when specified, overrides the image used to run Config Reloader. Specify the image along with a tag. You still need to set the version to ensure Grafana Agent Operator knows which version of Grafana Agent is being configured.
`imagePullSecrets` []Kubernetes core/v1.LocalObjectReference	ImagePullSecrets holds an optional list of references to Secrets within the same namespace used for pulling the Grafana Agent image from registries. More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
`storage` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.StorageSpec	Storage spec to specify how storage will be used.
`volumes` []Kubernetes core/v1.Volume	Volumes allows configuration of additional volumes on the output StatefulSet definition. The volumes specified are appended to other volumes that are generated as a result of StorageSpec objects.
`volumeMounts` []Kubernetes core/v1.VolumeMount	VolumeMounts lets you configure additional VolumeMounts on the output StatefulSet definition. Specified VolumeMounts are appended to other VolumeMounts generated as a result of StorageSpec objects in the Grafana Agent container.
`resources` Kubernetes core/v1.ResourceRequirements	Resources holds requests and limits for individual pods.
`nodeSelector` map[string]string	NodeSelector defines which nodes pods should be scheduling on.
`serviceAccountName` string	ServiceAccountName is the name of the ServiceAccount to use for running Grafana Agent pods.
`secrets` []string	Secrets is a list of secrets in the same namespace as the GrafanaAgent object which will be mounted into each running Grafana Agent pod. The secrets are mounted into /var/lib/grafana-agent/extra-secrets/<secret-name>.
`configMaps` []string	ConfigMaps is a list of config maps in the same namespace as the GrafanaAgent object which will be mounted into each running Grafana Agent pod. The ConfigMaps are mounted into /var/lib/grafana-agent/extra-configmaps/<configmap-name>.
`affinity` Kubernetes core/v1.Affinity	Affinity, if specified, controls pod scheduling constraints.
`tolerations` []Kubernetes core/v1.Toleration	Tolerations, if specified, controls the pod's tolerations.
`topologySpreadConstraints` []Kubernetes core/v1.TopologySpreadConstraint	TopologySpreadConstraints, if specified, controls the pod's topology spread constraints.
`securityContext` Kubernetes core/v1.PodSecurityContext	SecurityContext holds pod-level security attributes and common container settings. When unspecified, defaults to the default PodSecurityContext.
`containers` []Kubernetes core/v1.Container	Containers lets you inject additional containers or modify operator-generated containers. This can be used to add an authentication proxy to a Grafana Agent pod or to change the behavior of an operator-generated container. Containers described here modify an operator-generated container if they share the same name and if modifications are done via a strategic merge patch. The current container names are: `grafana-agent` and `config-reloader`. Overriding containers is entirely outside the scope of what the Grafana Agent team supports and by doing so, you accept that this behavior may break at any time without notice.
`initContainers` []Kubernetes core/v1.Container	InitContainers let you add initContainers to the pod definition. These can be used to, for example, fetch secrets for injection into the Grafana Agent configuration from external sources. Errors during the execution of an initContainer cause the pod to restart. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ Using initContainers for any use case other than secret fetching is entirely outside the scope of what the Grafana Agent maintainers support and by doing so, you accept that this behavior may break at any time without notice.
`priorityClassName` string	PriorityClassName is the priority class assigned to pods.
`runtimeClassName` string	RuntimeClassName is the runtime class assigned to pods.
`portName` string	Port name used for the pods and governing service. This defaults to agent-metrics.
`metrics` MetricsSubsystemSpec	Metrics controls the metrics subsystem of the Agent and settings unique to metrics-specific pods that are deployed.
`logs` LogsSubsystemSpec	Logs controls the logging subsystem of the Agent and settings unique to logging-specific pods that are deployed.
`integrations` IntegrationsSubsystemSpec	Integrations controls the integration subsystem of the Agent and settings unique to deployed integration-specific pods.
`enableConfigReadAPI` bool	enableConfigReadAPI enables the read API for viewing the currently running config port 8080 on the agent. +kubebuilder:default=false
`disableReporting` bool	disableReporting disables reporting of enabled feature flags to Grafana. +kubebuilder:default=false
`disableSupportBundle` bool	disableSupportBundle disables the generation of support bundles. +kubebuilder:default=false

IntegrationsDeployment

(Appears on:Deployment) IntegrationsDeployment is a set of discovered resources relative to an IntegrationsDeployment.

Fields

Field	Description
apiVersion	string `monitoring.grafana.com/v1alpha1`
kind	string `IntegrationsDeployment`
`Instance` Integration

LogsDeployment

(Appears on:Deployment) LogsDeployment is a set of discovered resources relative to a LogsInstance.

Fields

Field	Description
apiVersion	string `monitoring.grafana.com/v1alpha1`
kind	string `LogsDeployment`
`Instance` LogsInstance
`PodLogs` []PodLogs

MetricsDeployment

(Appears on:Deployment) MetricsDeployment is a set of discovered resources relative to a MetricsInstance.

Fields

Field	Description
apiVersion	string `monitoring.grafana.com/v1alpha1`
kind	string `MetricsDeployment`
`Instance` MetricsInstance
`ServiceMonitors` []github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.ServiceMonitor
`PodMonitors` []github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.PodMonitor
`Probes` []github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.Probe

CRIStageSpec

(Appears on:PipelineStageSpec) CRIStageSpec is a parsing stage that reads log lines using the standard CRI logging format. It needs no defined fields.

DockerStageSpec

(Appears on:PipelineStageSpec) DockerStageSpec is a parsing stage that reads log lines using the standard Docker logging format. It needs no defined fields.

DropStageSpec

(Appears on:PipelineStageSpec) DropStageSpec is a filtering stage that lets you drop certain logs.

Fields

Field	Description
`source` string	Name from the extract data to parse. If empty, uses the log message.
`expression` string	RE2 regular expression. If source is provided, the regex attempts to match the source. If no source is provided, then the regex attempts to attach the log line. If the provided regex matches the log line or a provided source, the line is dropped.
`value` string	Value can only be specified when source is specified. If the value provided is an exact match for the given source then the line will be dropped. Mutually exclusive with expression.
`olderThan` string	OlderThan will be parsed as a Go duration. If the log line's timestamp is older than the current time minus the provided duration, it will be dropped.
`longerThan` string	LongerThan will drop a log line if it its content is longer than this value (in bytes). Can be expressed as an integer (8192) or a number with a suffix (8kb).
`dropCounterReason` string	Every time a log line is dropped, the metric logentry_dropped_lines_total is incremented. A "reason" label is added, and can be customized by providing a custom value here. Defaults to "drop_stage".

GrafanaAgentSpec

(Appears on:GrafanaAgent) GrafanaAgentSpec is a specification of the desired behavior of the Grafana Agent cluster.

Fields

Field	Description
`logLevel` string	LogLevel controls the log level of the generated pods. Defaults to "info" if not set.
`logFormat` string	LogFormat controls the logging format of the generated pods. Defaults to "logfmt" if not set.
`apiServer` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.APIServerConfig	APIServerConfig lets you specify a host and auth methods to access the Kubernetes API server. If left empty, the Agent assumes that it is running inside of the cluster and will discover API servers automatically and use the pod's CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount.
`podMetadata` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.EmbeddedObjectMetadata	PodMetadata configures Labels and Annotations which are propagated to created Grafana Agent pods.
`version` string	Version of Grafana Agent to be deployed.
`paused` bool	Paused prevents actions except for deletion to be performed on the underlying managed objects.
`image` string	Image, when specified, overrides the image used to run Agent. Specify the image along with a tag. You still need to set the version to ensure Grafana Agent Operator knows which version of Grafana Agent is being configured.
`configReloaderVersion` string	Version of Config Reloader to be deployed.
`configReloaderImage` string	Image, when specified, overrides the image used to run Config Reloader. Specify the image along with a tag. You still need to set the version to ensure Grafana Agent Operator knows which version of Grafana Agent is being configured.
`imagePullSecrets` []Kubernetes core/v1.LocalObjectReference	ImagePullSecrets holds an optional list of references to Secrets within the same namespace used for pulling the Grafana Agent image from registries. More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
`storage` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.StorageSpec	Storage spec to specify how storage will be used.
`volumes` []Kubernetes core/v1.Volume	Volumes allows configuration of additional volumes on the output StatefulSet definition. The volumes specified are appended to other volumes that are generated as a result of StorageSpec objects.
`volumeMounts` []Kubernetes core/v1.VolumeMount	VolumeMounts lets you configure additional VolumeMounts on the output StatefulSet definition. Specified VolumeMounts are appended to other VolumeMounts generated as a result of StorageSpec objects in the Grafana Agent container.
`resources` Kubernetes core/v1.ResourceRequirements	Resources holds requests and limits for individual pods.
`nodeSelector` map[string]string	NodeSelector defines which nodes pods should be scheduling on.
`serviceAccountName` string	ServiceAccountName is the name of the ServiceAccount to use for running Grafana Agent pods.
`secrets` []string	Secrets is a list of secrets in the same namespace as the GrafanaAgent object which will be mounted into each running Grafana Agent pod. The secrets are mounted into /var/lib/grafana-agent/extra-secrets/<secret-name>.
`configMaps` []string	ConfigMaps is a list of config maps in the same namespace as the GrafanaAgent object which will be mounted into each running Grafana Agent pod. The ConfigMaps are mounted into /var/lib/grafana-agent/extra-configmaps/<configmap-name>.
`affinity` Kubernetes core/v1.Affinity	Affinity, if specified, controls pod scheduling constraints.
`tolerations` []Kubernetes core/v1.Toleration	Tolerations, if specified, controls the pod's tolerations.
`topologySpreadConstraints` []Kubernetes core/v1.TopologySpreadConstraint	TopologySpreadConstraints, if specified, controls the pod's topology spread constraints.
`securityContext` Kubernetes core/v1.PodSecurityContext	SecurityContext holds pod-level security attributes and common container settings. When unspecified, defaults to the default PodSecurityContext.
`containers` []Kubernetes core/v1.Container	Containers lets you inject additional containers or modify operator-generated containers. This can be used to add an authentication proxy to a Grafana Agent pod or to change the behavior of an operator-generated container. Containers described here modify an operator-generated container if they share the same name and if modifications are done via a strategic merge patch. The current container names are: `grafana-agent` and `config-reloader`. Overriding containers is entirely outside the scope of what the Grafana Agent team supports and by doing so, you accept that this behavior may break at any time without notice.
`initContainers` []Kubernetes core/v1.Container	InitContainers let you add initContainers to the pod definition. These can be used to, for example, fetch secrets for injection into the Grafana Agent configuration from external sources. Errors during the execution of an initContainer cause the pod to restart. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ Using initContainers for any use case other than secret fetching is entirely outside the scope of what the Grafana Agent maintainers support and by doing so, you accept that this behavior may break at any time without notice.
`priorityClassName` string	PriorityClassName is the priority class assigned to pods.
`runtimeClassName` string	RuntimeClassName is the runtime class assigned to pods.
`portName` string	Port name used for the pods and governing service. This defaults to agent-metrics.
`metrics` MetricsSubsystemSpec	Metrics controls the metrics subsystem of the Agent and settings unique to metrics-specific pods that are deployed.
`logs` LogsSubsystemSpec	Logs controls the logging subsystem of the Agent and settings unique to logging-specific pods that are deployed.
`integrations` IntegrationsSubsystemSpec	Integrations controls the integration subsystem of the Agent and settings unique to deployed integration-specific pods.
`enableConfigReadAPI` bool	enableConfigReadAPI enables the read API for viewing the currently running config port 8080 on the agent. +kubebuilder:default=false
`disableReporting` bool	disableReporting disables reporting of enabled feature flags to Grafana. +kubebuilder:default=false
`disableSupportBundle` bool	disableSupportBundle disables the generation of support bundles. +kubebuilder:default=false

Integration

(Appears on:IntegrationsDeployment) Integration runs a single Grafana Agent integration. Integrations that generate telemetry must be configured to send that telemetry somewhere, such as autoscrape for exporter-based integrations. Integrations have access to the LogsInstances and MetricsInstances in the same GrafanaAgent resource set, referenced by the <namespace>/<name> of the Instance resource. For example, if there is a default/production MetricsInstance, you can configure a supported integration's autoscrape block with: autoscrape: enable: true metrics_instance: default/production There is currently no way for telemetry created by an Operator-managed integration to be collected from outside of the integration itself.

Fields

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` IntegrationSpec	Specifies the desired behavior of the Integration.
`name` string	Name of the integration to run (e.g., "node_exporter", "mysqld_exporter").
`type` IntegrationType	Type informs Grafana Agent Operator about how to manage the integration being configured.
`config` k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.JSON	The configuration for the named integration. Note that Integrations are deployed with the integrations-next feature flag, which has different common settings: https://grafana.com/docs/agent/latest/configuration/integrations/integrations-next/
`volumes` []Kubernetes core/v1.Volume	An extra list of Volumes to be associated with the Grafana Agent pods running this integration. Volume names are mutated to be unique across all Integrations. Note that the specified volumes should be able to tolerate existing on multiple pods at once when type is daemonset. Don't use volumes for loading Secrets or ConfigMaps from the same namespace as the Integration; use the Secrets and ConfigMaps fields instead.
`volumeMounts` []Kubernetes core/v1.VolumeMount	An extra list of VolumeMounts to be associated with the Grafana Agent pods running this integration. VolumeMount names are mutated to be unique across all used IntegrationSpecs. Mount paths should include the namespace/name of the Integration CR to avoid potentially colliding with other resources.
`secrets` []Kubernetes core/v1.SecretKeySelector	An extra list of keys from Secrets in the same namespace as the Integration which will be mounted into the Grafana Agent pod running this Integration. Secrets will be mounted at /etc/grafana-agent/integrations/secrets/<secret_namespace>/<secret_name>/<key>.
`configMaps` []Kubernetes core/v1.ConfigMapKeySelector	An extra list of keys from ConfigMaps in the same namespace as the Integration which will be mounted into the Grafana Agent pod running this Integration. ConfigMaps are mounted at /etc/grafana-agent/integrations/configMaps/<configmap_namespace>/<configmap_name>/<key>.

IntegrationSpec

(Appears on:Integration) IntegrationSpec specifies the desired behavior of a metrics integration.

Fields

Field	Description
`name` string	Name of the integration to run (e.g., "node_exporter", "mysqld_exporter").
`type` IntegrationType	Type informs Grafana Agent Operator about how to manage the integration being configured.
`config` k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.JSON	The configuration for the named integration. Note that Integrations are deployed with the integrations-next feature flag, which has different common settings: https://grafana.com/docs/agent/latest/configuration/integrations/integrations-next/
`volumes` []Kubernetes core/v1.Volume	An extra list of Volumes to be associated with the Grafana Agent pods running this integration. Volume names are mutated to be unique across all Integrations. Note that the specified volumes should be able to tolerate existing on multiple pods at once when type is daemonset. Don't use volumes for loading Secrets or ConfigMaps from the same namespace as the Integration; use the Secrets and ConfigMaps fields instead.
`volumeMounts` []Kubernetes core/v1.VolumeMount	An extra list of VolumeMounts to be associated with the Grafana Agent pods running this integration. VolumeMount names are mutated to be unique across all used IntegrationSpecs. Mount paths should include the namespace/name of the Integration CR to avoid potentially colliding with other resources.
`secrets` []Kubernetes core/v1.SecretKeySelector	An extra list of keys from Secrets in the same namespace as the Integration which will be mounted into the Grafana Agent pod running this Integration. Secrets will be mounted at /etc/grafana-agent/integrations/secrets/<secret_namespace>/<secret_name>/<key>.
`configMaps` []Kubernetes core/v1.ConfigMapKeySelector	An extra list of keys from ConfigMaps in the same namespace as the Integration which will be mounted into the Grafana Agent pod running this Integration. ConfigMaps are mounted at /etc/grafana-agent/integrations/configMaps/<configmap_namespace>/<configmap_name>/<key>.

IntegrationType

(Appears on:IntegrationSpec) IntegrationType determines specific behaviors of a configured integration.

Fields

Field	Description
`allNodes` bool	When true, the configured integration should be run on every Node in the cluster. This is required for Integrations that generate Node-specific metrics like node_exporter, otherwise it must be false to avoid generating duplicate metrics.
`unique` bool	Whether this integration can only be defined once for a Grafana Agent process, such as statsd_exporter. It is invalid for a GrafanaAgent to discover multiple unique Integrations with the same Integration name (i.e., a single GrafanaAgent cannot deploy two statsd_exporters).

IntegrationsSubsystemSpec

(Appears on:GrafanaAgentSpec) IntegrationsSubsystemSpec defines global settings to apply across the integrations subsystem.

Fields

Field	Description
`selector` Kubernetes meta/v1.LabelSelector	Label selector to find Integration resources to run. When nil, no integration resources will be defined.
`namespaceSelector` Kubernetes meta/v1.LabelSelector	Label selector for namespaces to search when discovering integration resources. If nil, integration resources are only discovered in the namespace of the GrafanaAgent resource. Set to `{}` to search all namespaces.

JSONStageSpec

(Appears on:PipelineStageSpec) JSONStageSpec is a parsing stage that reads the log line as JSON and accepts JMESPath expressions to extract data.

Fields

Field	Description
`source` string	Name from the extracted data to parse as JSON. If empty, uses entire log message.
`expressions` map[string]string	Set of the key/value pairs of JMESPath expressions. The key will be the key in the extracted data while the expression will be the value, evaluated as a JMESPath from the source data. Literal JMESPath expressions can be used by wrapping a key in double quotes, which then must be wrapped again in single quotes in YAML so they get passed to the JMESPath parser.

LimitStageSpec

(Appears on:PipelineStageSpec) The limit stage is a rate-limiting stage that throttles logs based on several options.

Fields

Field	Description
`rate` int	The rate limit in lines per second that Promtail will push to Loki.
`burst` int	The cap in the quantity of burst lines that Promtail will push to Loki.
`drop` bool	When drop is true, log lines that exceed the current rate limit are discarded. When drop is false, log lines that exceed the current rate limit wait to enter the back pressure mode. Defaults to false.

LogsBackoffConfigSpec

(Appears on:LogsClientSpec) LogsBackoffConfigSpec configures timing for retrying failed requests.

Fields

Field	Description
`minPeriod` string	Initial backoff time between retries. Time between retries is increased exponentially.
`maxPeriod` string	Maximum backoff time between retries.
`maxRetries` int	Maximum number of retries to perform before giving up a request.

LogsClientSpec

(Appears on:LogsInstanceSpec, LogsSubsystemSpec) LogsClientSpec defines the client integration for logs, indicating which Loki server to send logs to.

Fields

Field	Description
`url` string	URL is the URL where Loki is listening. Must be a full HTTP URL, including protocol. Required. Example: https://logs-prod-us-central1.grafana.net/loki/api/v1/push.
`tenantId` string	Tenant ID used by default to push logs to Loki. If omitted assumes remote Loki is running in single-tenant mode or an authentication layer is used to inject an X-Scope-OrgID header.
`batchWait` string	Maximum amount of time to wait before sending a batch, even if that batch isn't full.
`batchSize` int	Maximum batch size (in bytes) of logs to accumulate before sending the batch to Loki.
`basicAuth` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.BasicAuth	BasicAuth for the Loki server.
`oauth2` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.OAuth2	Oauth2 for URL
`bearerToken` string	BearerToken used for remote_write.
`bearerTokenFile` string	BearerTokenFile used to read bearer token.
`proxyUrl` string	ProxyURL to proxy requests through. Optional.
`tlsConfig` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.TLSConfig	TLSConfig to use for the client. Only used when the protocol of the URL is https.
`backoffConfig` LogsBackoffConfigSpec	Configures how to retry requests to Loki when a request fails. Defaults to a minPeriod of 500ms, maxPeriod of 5m, and maxRetries of 10.
`externalLabels` map[string]string	ExternalLabels are labels to add to any time series when sending data to Loki.
`timeout` string	Maximum time to wait for a server to respond to a request.

LogsInstance

(Appears on:LogsDeployment) LogsInstance controls an individual logs instance within a Grafana Agent deployment.

Fields

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` LogsInstanceSpec	Spec holds the specification of the desired behavior for the logs instance.
`clients` []LogsClientSpec	Clients controls where logs are written to for this instance.
`podLogsSelector` Kubernetes meta/v1.LabelSelector	Determines which PodLogs should be selected for including in this instance.
`podLogsNamespaceSelector` Kubernetes meta/v1.LabelSelector	Set of labels to determine which namespaces should be watched for PodLogs. If not provided, checks only namespace of the instance.
`additionalScrapeConfigs` Kubernetes core/v1.SecretKeySelector	AdditionalScrapeConfigs allows specifying a key of a Secret containing additional Grafana Agent logging scrape configurations. Scrape configurations specified are appended to the configurations generated by the Grafana Agent Operator. Job configurations specified must have the form as specified in the official Promtail documentation: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#scrape_configs As scrape configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of Grafana Agent. It is advised to review both Grafana Agent and Promtail release notes to ensure that no incompatible scrape configs are going to break Grafana Agent after the upgrade.
`targetConfig` LogsTargetConfigSpec	Configures how tailed targets are watched.

LogsInstanceSpec

(Appears on:LogsInstance) LogsInstanceSpec controls how an individual instance will be used to discover LogMonitors.

Fields

Field	Description
`clients` []LogsClientSpec	Clients controls where logs are written to for this instance.
`podLogsSelector` Kubernetes meta/v1.LabelSelector	Determines which PodLogs should be selected for including in this instance.
`podLogsNamespaceSelector` Kubernetes meta/v1.LabelSelector	Set of labels to determine which namespaces should be watched for PodLogs. If not provided, checks only namespace of the instance.
`additionalScrapeConfigs` Kubernetes core/v1.SecretKeySelector	AdditionalScrapeConfigs allows specifying a key of a Secret containing additional Grafana Agent logging scrape configurations. Scrape configurations specified are appended to the configurations generated by the Grafana Agent Operator. Job configurations specified must have the form as specified in the official Promtail documentation: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#scrape_configs As scrape configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of Grafana Agent. It is advised to review both Grafana Agent and Promtail release notes to ensure that no incompatible scrape configs are going to break Grafana Agent after the upgrade.
`targetConfig` LogsTargetConfigSpec	Configures how tailed targets are watched.

LogsSubsystemSpec

(Appears on:GrafanaAgentSpec) LogsSubsystemSpec defines global settings to apply across the logging subsystem.

Fields

Field	Description
`clients` []LogsClientSpec	A global set of clients to use when a discovered LogsInstance does not have any clients defined.
`logsExternalLabelName` string	LogsExternalLabelName is the name of the external label used to denote Grafana Agent cluster. Defaults to "cluster." External label will not be added when value is set to the empty string.
`instanceSelector` Kubernetes meta/v1.LabelSelector	InstanceSelector determines which LogInstances should be selected for running. Each instance runs its own set of Prometheus components, including service discovery, scraping, and remote_write.
`instanceNamespaceSelector` Kubernetes meta/v1.LabelSelector	InstanceNamespaceSelector are the set of labels to determine which namespaces to watch for LogInstances. If not provided, only checks own namespace.
`ignoreNamespaceSelectors` bool	IgnoreNamespaceSelectors, if true, will ignore NamespaceSelector settings from the PodLogs configs, and they will only discover endpoints within their current namespace.
`enforcedNamespaceLabel` string	EnforcedNamespaceLabel enforces adding a namespace label of origin for each metric that is user-created. The label value will always be the namespace of the object that is being created.

LogsTargetConfigSpec

(Appears on:LogsInstanceSpec) LogsTargetConfigSpec configures how tailed targets are watched.

Fields

Field	Description
`syncPeriod` string	Period to resync directories being watched and files being tailed to discover new ones or stop watching removed ones.

MatchStageSpec

(Appears on:PipelineStageSpec) MatchStageSpec is a filtering stage that conditionally applies a set of stages or drop entries when a log entry matches a configurable LogQL stream selector and filter expressions.

Fields

Field	Description
`selector` string	LogQL stream selector and filter expressions. Required.
`pipelineName` string	Names the pipeline. When defined, creates an additional label in the pipeline_duration_seconds histogram, where the value is concatenated with job_name using an underscore.
`action` string	Determines what action is taken when the selector matches the log line. Can be keep or drop. Defaults to keep. When set to drop, entries are dropped and no later metrics are recorded. Stages must be empty when dropping metrics.
`dropCounterReason` string	Every time a log line is dropped, the metric logentry_dropped_lines_total is incremented. A "reason" label is added, and can be customized by providing a custom value here. Defaults to "match_stage."
`stages` string	Nested set of pipeline stages to execute when action is keep and the log line matches selector. An example value for stages may be: stages:

MetadataConfig

(Appears on:RemoteWriteSpec) MetadataConfig configures the sending of series metadata to remote storage.

Fields

Field	Description
`send` bool	Send enables metric metadata to be sent to remote storage.
`sendInterval` string	SendInterval controls how frequently metric metadata is sent to remote storage.

MetricsInstance

(Appears on:MetricsDeployment) MetricsInstance controls an individual Metrics instance within a Grafana Agent deployment.

Fields

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` MetricsInstanceSpec	Spec holds the specification of the desired behavior for the Metrics instance.
`walTruncateFrequency` string	WALTruncateFrequency specifies how frequently to run the WAL truncation process. Higher values cause the WAL to increase and for old series to stay in the WAL longer, but reduces the chance of data loss when remote_write fails for longer than the given frequency.
`minWALTime` string	MinWALTime is the minimum amount of time that series and samples can exist in the WAL before being considered for deletion.
`maxWALTime` string	MaxWALTime is the maximum amount of time that series and samples can exist in the WAL before being forcibly deleted.
`remoteFlushDeadline` string	RemoteFlushDeadline is the deadline for flushing data when an instance shuts down.
`writeStaleOnShutdown` bool	WriteStaleOnShutdown writes staleness markers on shutdown for all series.
`serviceMonitorSelector` Kubernetes meta/v1.LabelSelector	ServiceMonitorSelector determines which ServiceMonitors to select for target discovery.
`serviceMonitorNamespaceSelector` Kubernetes meta/v1.LabelSelector	ServiceMonitorNamespaceSelector is the set of labels that determine which namespaces to watch for ServiceMonitor discovery. If nil, it only checks its own namespace.
`podMonitorSelector` Kubernetes meta/v1.LabelSelector	PodMonitorSelector determines which PodMonitors to selected for target discovery. Experimental.
`podMonitorNamespaceSelector` Kubernetes meta/v1.LabelSelector	PodMonitorNamespaceSelector are the set of labels to determine which namespaces to watch for PodMonitor discovery. If nil, it only checks its own namespace.
`probeSelector` Kubernetes meta/v1.LabelSelector	ProbeSelector determines which Probes to select for target discovery.
`probeNamespaceSelector` Kubernetes meta/v1.LabelSelector	ProbeNamespaceSelector is the set of labels that determines which namespaces to watch for Probe discovery. If nil, it only checks own namespace.
`remoteWrite` []RemoteWriteSpec	RemoteWrite controls remote_write settings for this instance.
`additionalScrapeConfigs` Kubernetes core/v1.SecretKeySelector	AdditionalScrapeConfigs lets you specify a key of a Secret containing additional Grafana Agent Prometheus scrape configurations. The specified scrape configurations are appended to the configurations generated by Grafana Agent Operator. Specified job configurations must have the form specified in the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config. As scrape configs are appended, you must make sure the configuration is still valid. Note that it's possible that this feature will break future upgrades of Grafana Agent. Review both Grafana Agent and Prometheus release notes to ensure that no incompatible scrape configs will break Grafana Agent after the upgrade.

MetricsInstanceSpec

(Appears on:MetricsInstance) MetricsInstanceSpec controls how an individual instance is used to discover PodMonitors.

Fields

Field	Description
`walTruncateFrequency` string	WALTruncateFrequency specifies how frequently to run the WAL truncation process. Higher values cause the WAL to increase and for old series to stay in the WAL longer, but reduces the chance of data loss when remote_write fails for longer than the given frequency.
`minWALTime` string	MinWALTime is the minimum amount of time that series and samples can exist in the WAL before being considered for deletion.
`maxWALTime` string	MaxWALTime is the maximum amount of time that series and samples can exist in the WAL before being forcibly deleted.
`remoteFlushDeadline` string	RemoteFlushDeadline is the deadline for flushing data when an instance shuts down.
`writeStaleOnShutdown` bool	WriteStaleOnShutdown writes staleness markers on shutdown for all series.
`serviceMonitorSelector` Kubernetes meta/v1.LabelSelector	ServiceMonitorSelector determines which ServiceMonitors to select for target discovery.
`serviceMonitorNamespaceSelector` Kubernetes meta/v1.LabelSelector	ServiceMonitorNamespaceSelector is the set of labels that determine which namespaces to watch for ServiceMonitor discovery. If nil, it only checks its own namespace.
`podMonitorSelector` Kubernetes meta/v1.LabelSelector	PodMonitorSelector determines which PodMonitors to selected for target discovery. Experimental.
`podMonitorNamespaceSelector` Kubernetes meta/v1.LabelSelector	PodMonitorNamespaceSelector are the set of labels to determine which namespaces to watch for PodMonitor discovery. If nil, it only checks its own namespace.
`probeSelector` Kubernetes meta/v1.LabelSelector	ProbeSelector determines which Probes to select for target discovery.
`probeNamespaceSelector` Kubernetes meta/v1.LabelSelector	ProbeNamespaceSelector is the set of labels that determines which namespaces to watch for Probe discovery. If nil, it only checks own namespace.
`remoteWrite` []RemoteWriteSpec	RemoteWrite controls remote_write settings for this instance.
`additionalScrapeConfigs` Kubernetes core/v1.SecretKeySelector	AdditionalScrapeConfigs lets you specify a key of a Secret containing additional Grafana Agent Prometheus scrape configurations. The specified scrape configurations are appended to the configurations generated by Grafana Agent Operator. Specified job configurations must have the form specified in the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config. As scrape configs are appended, you must make sure the configuration is still valid. Note that it's possible that this feature will break future upgrades of Grafana Agent. Review both Grafana Agent and Prometheus release notes to ensure that no incompatible scrape configs will break Grafana Agent after the upgrade.

MetricsStageSpec

(Appears on:PipelineStageSpec) MetricsStageSpec is an action stage that allows for defining and updating metrics based on data from the extracted map. Created metrics are not pushed to Loki or Prometheus and are instead exposed via the /metrics endpoint of the Grafana Agent pod. The Grafana Agent Operator should be configured with a MetricsInstance that discovers the logging DaemonSet to collect metrics created by this stage.

Fields

Field	Description
`type` string	The metric type to create. Must be one of counter, gauge, histogram. Required.
`description` string	Sets the description for the created metric.
`prefix` string	Sets the custom prefix name for the metric. Defaults to "promtail_custom_".
`source` string	Key from the extracted data map to use for the metric. Defaults to the metrics name if not present.
`maxIdleDuration` string	Label values on metrics are dynamic which can cause exported metrics to go stale. To prevent unbounded cardinality, any metrics not updated within MaxIdleDuration are removed. Must be greater or equal to 1s. Defaults to 5m.
`matchAll` bool	If true, all log lines are counted without attempting to match the source to the extracted map. Mutually exclusive with value. Only valid for type: counter.
`countEntryBytes` bool	If true all log line bytes are counted. Can only be set with matchAll: true and action: add. Only valid for type: counter.
`value` string	Filters down source data and only changes the metric if the targeted value matches the provided string exactly. If not present, all data matches.
`action` string	The action to take against the metric. Required. Must be either "inc" or "add" for type: counter or type: histogram. When type: gauge, must be one of "set", "inc", "dec", "add", or "sub". "add", "set", or "sub" requires the extracted value to be convertible to a positive float.
`buckets` []string	Buckets to create. Bucket values must be convertible to float64s. Extremely large or small numbers are subject to some loss of precision. Only valid for type: histogram.

MetricsSubsystemSpec

(Appears on:GrafanaAgentSpec) MetricsSubsystemSpec defines global settings to apply across the Metrics subsystem.

Fields

Field	Description
`remoteWrite` []RemoteWriteSpec	RemoteWrite controls default remote_write settings for all instances. If an instance does not provide its own RemoteWrite settings, these will be used instead.
`replicas` int32	Replicas of each shard to deploy for metrics pods. Number of replicas multiplied by the number of shards is the total number of pods created.
`shards` int32	Shards to distribute targets onto. Number of replicas multiplied by the number of shards is the total number of pods created. Note that scaling down shards does not reshard data onto remaining instances; it must be manually moved. Increasing shards does not reshard data either, but it will continue to be available from the same instances. Sharding is performed on the content of the address target meta-label.
`replicaExternalLabelName` string	ReplicaExternalLabelName is the name of the metrics external label used to denote the replica name. Defaults to replica. The external label is not added when the value is set to the empty string.
`metricsExternalLabelName` string	MetricsExternalLabelName is the name of the external label used to denote Grafana Agent cluster. Defaults to "cluster." The external label is not added when the value is set to the empty string.
`scrapeInterval` string	ScrapeInterval is the time between consecutive scrapes.
`scrapeTimeout` string	ScrapeTimeout is the time to wait for a target to respond before marking a scrape as failed.
`externalLabels` map[string]string	ExternalLabels are labels to add to any time series when sending data over remote_write.
`arbitraryFSAccessThroughSMs` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.ArbitraryFSAccessThroughSMsConfig	ArbitraryFSAccessThroughSMs configures whether configuration based on a ServiceMonitor can access arbitrary files on the file system of the Grafana Agent container, e.g., bearer token files.
`overrideHonorLabels` bool	OverrideHonorLabels, if true, overrides all configured honor_labels read from ServiceMonitor or PodMonitor and sets them to false.
`overrideHonorTimestamps` bool	OverrideHonorTimestamps allows global enforcement for honoring timestamps in all scrape configs.
`ignoreNamespaceSelectors` bool	IgnoreNamespaceSelectors, if true, ignores NamespaceSelector settings from the PodMonitor and ServiceMonitor configs, so that they only discover endpoints within their current namespace.
`enforcedNamespaceLabel` string	EnforcedNamespaceLabel enforces adding a namespace label of origin for each metric that is user-created. The label value is always the namespace of the object that is being created.
`enforcedSampleLimit` uint64	EnforcedSampleLimit defines a global limit on the number of scraped samples that are accepted. This overrides any SampleLimit set per ServiceMonitor and/or PodMonitor. It is meant to be used by admins to enforce the SampleLimit to keep the overall number of samples and series under the desired limit. Note that if a SampleLimit from a ServiceMonitor or PodMonitor is lower, that value is used instead.
`enforcedTargetLimit` uint64	EnforcedTargetLimit defines a global limit on the number of scraped targets. This overrides any TargetLimit set per ServiceMonitor and/or PodMonitor. It is meant to be used by admins to enforce the TargetLimit to keep the overall number of targets under the desired limit. Note that if a TargetLimit from a ServiceMonitor or PodMonitor is higher, that value is used instead.
`instanceSelector` Kubernetes meta/v1.LabelSelector	InstanceSelector determines which MetricsInstances should be selected for running. Each instance runs its own set of Metrics components, including service discovery, scraping, and remote_write.
`instanceNamespaceSelector` Kubernetes meta/v1.LabelSelector	InstanceNamespaceSelector is the set of labels that determines which namespaces to watch for MetricsInstances. If not provided, it only checks its own namespace.

MultilineStageSpec

(Appears on:PipelineStageSpec) MultilineStageSpec merges multiple lines into a multiline block before passing it on to the next stage in the pipeline.

Fields

Field	Description
`firstLine` string	RE2 regular expression. Creates a new multiline block when matched. Required.
`maxWaitTime` string	Maximum time to wait before passing on the multiline block to the next stage if no new lines are received. Defaults to 3s.
`maxLines` int	Maximum number of lines a block can have. A new block is started if the number of lines surpasses this value. Defaults to 128.

ObjectSelector

ObjectSelector is a set of selectors to use for finding an object in the resource hierarchy. When NamespaceSelector is nil, search for objects directly in the ParentNamespace.

Fields

Field	Description
`ObjectType` sigs.k8s.io/controller-runtime/pkg/client.Object
`ParentNamespace` string
`NamespaceSelector` Kubernetes meta/v1.LabelSelector
`Labels` Kubernetes meta/v1.LabelSelector

OutputStageSpec

(Appears on:PipelineStageSpec) OutputStageSpec is an action stage that takes data from the extracted map and changes the log line that will be sent to Loki.

Fields

Field	Description
`source` string	Name from extract data to use for the log entry. Required.

PackStageSpec

(Appears on:PipelineStageSpec) PackStageSpec is a transform stage that lets you embed extracted values and labels into the log line by packing the log line and labels inside of a JSON object.

Fields

Field	Description
`labels` []string	Name from extracted data or line labels. Required. Labels provided here are automatically removed from output labels.
`ingestTimestamp` bool	If the resulting log line should use any existing timestamp or use time.Now() when the line was created. Set to true when combining several log streams from different containers to avoid out of order errors.

PipelineStageSpec

(Appears on:PodLogsSpec) PipelineStageSpec defines an individual pipeline stage. Each stage type is mutually exclusive and no more than one may be set per stage. More information on pipelines can be found in the Promtail documentation: https://grafana.com/docs/loki/latest/clients/promtail/pipelines/

Fields

Field	Description
`cri` CRIStageSpec	CRI is a parsing stage that reads log lines using the standard CRI logging format. Supply cri: {} to enable.
`docker` DockerStageSpec	Docker is a parsing stage that reads log lines using the standard Docker logging format. Supply docker: {} to enable.
`drop` DropStageSpec	Drop is a filtering stage that lets you drop certain logs.
`json` JSONStageSpec	JSON is a parsing stage that reads the log line as JSON and accepts JMESPath expressions to extract data. Information on JMESPath: http://jmespath.org/
`labelAllow` []string	LabelAllow is an action stage that only allows the provided labels to be included in the label set that is sent to Loki with the log entry.
`labelDrop` []string	LabelDrop is an action stage that drops labels from the label set that is sent to Loki with the log entry.
`labels` map[string]string	Labels is an action stage that takes data from the extracted map and modifies the label set that is sent to Loki with the log entry. The key is REQUIRED and represents the name for the label that will be created. Value is optional and will be the name from extracted data to use for the value of the label. If the value is not provided, it defaults to match the key.
`limit` LimitStageSpec	Limit is a rate-limiting stage that throttles logs based on several options.
`match` MatchStageSpec	Match is a filtering stage that conditionally applies a set of stages or drop entries when a log entry matches a configurable LogQL stream selector and filter expressions.
`metrics` map[string]github.com/grafana/agent/static/operator/apis/monitoring/v1alpha1.MetricsStageSpec	Metrics is an action stage that supports defining and updating metrics based on data from the extracted map. Created metrics are not pushed to Loki or Prometheus and are instead exposed via the /metrics endpoint of the Grafana Agent pod. The Grafana Agent Operator should be configured with a MetricsInstance that discovers the logging DaemonSet to collect metrics created by this stage.
`multiline` MultilineStageSpec	Multiline stage merges multiple lines into a multiline block before passing it on to the next stage in the pipeline.
`output` OutputStageSpec	Output stage is an action stage that takes data from the extracted map and changes the log line that will be sent to Loki.
`pack` PackStageSpec	Pack is a transform stage that lets you embed extracted values and labels into the log line by packing the log line and labels inside of a JSON object.
`regex` RegexStageSpec	Regex is a parsing stage that parses a log line using a regular expression. Named capture groups in the regex allows for adding data into the extracted map.
`replace` ReplaceStageSpec	Replace is a parsing stage that parses a log line using a regular expression and replaces the log line. Named capture groups in the regex allows for adding data into the extracted map.
`template` TemplateStageSpec	Template is a transform stage that manipulates the values in the extracted map using Go's template syntax.
`tenant` TenantStageSpec	Tenant is an action stage that sets the tenant ID for the log entry picking it from a field in the extracted data map. If the field is missing, the default LogsClientSpec.tenantId will be used.
`timestamp` TimestampStageSpec	Timestamp is an action stage that can change the timestamp of a log line before it is sent to Loki. If not present, the timestamp of a log line defaults to the time when the log line was read.

PodLogs

(Appears on:LogsDeployment) PodLogs defines how to collect logs for a pod.

Fields

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` PodLogsSpec	Spec holds the specification of the desired behavior for the PodLogs.
`jobLabel` string	The label to use to retrieve the job name from.
`podTargetLabels` []string	PodTargetLabels transfers labels on the Kubernetes Pod onto the target.
`selector` Kubernetes meta/v1.LabelSelector	Selector to select Pod objects. Required.
`namespaceSelector` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.NamespaceSelector	Selector to select which namespaces the Pod objects are discovered from.
`pipelineStages` []PipelineStageSpec	Pipeline stages for this pod. Pipeline stages support transforming and filtering log lines.
`relabelings` []github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.RelabelConfig	RelabelConfigs to apply to logs before delivering. Grafana Agent Operator automatically adds relabelings for a few standard Kubernetes fields and replaces original scrape job name with __tmp_logs_job_name. More info: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#relabel_configs

PodLogsSpec

(Appears on:PodLogs) PodLogsSpec defines how to collect logs for a pod.

Fields

Field	Description
`jobLabel` string	The label to use to retrieve the job name from.
`podTargetLabels` []string	PodTargetLabels transfers labels on the Kubernetes Pod onto the target.
`selector` Kubernetes meta/v1.LabelSelector	Selector to select Pod objects. Required.
`namespaceSelector` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.NamespaceSelector	Selector to select which namespaces the Pod objects are discovered from.
`pipelineStages` []PipelineStageSpec	Pipeline stages for this pod. Pipeline stages support transforming and filtering log lines.
`relabelings` []github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.RelabelConfig	RelabelConfigs to apply to logs before delivering. Grafana Agent Operator automatically adds relabelings for a few standard Kubernetes fields and replaces original scrape job name with __tmp_logs_job_name. More info: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#relabel_configs

QueueConfig

(Appears on:RemoteWriteSpec) QueueConfig allows the tuning of remote_write queue_config parameters.

Fields

Field	Description
`capacity` int	Capacity is the number of samples to buffer per shard before samples start being dropped.
`minShards` int	MinShards is the minimum number of shards, i.e., the amount of concurrency.
`maxShards` int	MaxShards is the maximum number of shards, i.e., the amount of concurrency.
`maxSamplesPerSend` int	MaxSamplesPerSend is the maximum number of samples per send.
`batchSendDeadline` string	BatchSendDeadline is the maximum time a sample will wait in the buffer.
`maxRetries` int	MaxRetries is the maximum number of times to retry a batch on recoverable errors.
`minBackoff` string	MinBackoff is the initial retry delay. MinBackoff is doubled for every retry.
`maxBackoff` string	MaxBackoff is the maximum retry delay.
`retryOnRateLimit` bool	RetryOnRateLimit retries requests when encountering rate limits.

RegexStageSpec

(Appears on:PipelineStageSpec) RegexStageSpec is a parsing stage that parses a log line using a regular expression. Named capture groups in the regex allows for adding data into the extracted map.

Fields

Field	Description
`source` string	Name from extracted data to parse. If empty, defaults to using the log message.
`expression` string	RE2 regular expression. Each capture group MUST be named. Required.

RemoteWriteSpec

(Appears on:MetricsInstanceSpec, MetricsSubsystemSpec) RemoteWriteSpec defines the remote_write configuration for Prometheus.

Fields

Field	Description
`name` string	Name of the remote_write queue. Must be unique if specified. The name is used in metrics and logging in order to differentiate queues.
`url` string	URL of the endpoint to send samples to.
`remoteTimeout` string	RemoteTimeout is the timeout for requests to the remote_write endpoint.
`headers` map[string]string	Headers is a set of custom HTTP headers to be sent along with each remote_write request. Be aware that any headers set by Grafana Agent itself can't be overwritten.
`writeRelabelConfigs` []github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.RelabelConfig	WriteRelabelConfigs holds relabel_configs to relabel samples before they are sent to the remote_write endpoint.
`basicAuth` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.BasicAuth	BasicAuth for the URL.
`oauth2` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.OAuth2	Oauth2 for URL
`bearerToken` string	BearerToken used for remote_write.
`bearerTokenFile` string	BearerTokenFile used to read bearer token.
`sigv4` SigV4Config	SigV4 configures SigV4-based authentication to the remote_write endpoint. SigV4-based authentication is used if SigV4 is defined, even with an empty object.
`tlsConfig` github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1.TLSConfig	TLSConfig to use for remote_write.
`proxyUrl` string	ProxyURL to proxy requests through. Optional.
`queueConfig` QueueConfig	QueueConfig allows tuning of the remote_write queue parameters.
`metadataConfig` MetadataConfig	MetadataConfig configures the sending of series metadata to remote storage.

ReplaceStageSpec

(Appears on:PipelineStageSpec) ReplaceStageSpec is a parsing stage that parses a log line using a regular expression and replaces the log line. Named capture groups in the regex allows for adding data into the extracted map.

Fields

Field	Description
`source` string	Name from extracted data to parse. If empty, defaults to using the log message.
`expression` string	RE2 regular expression. Each capture group MUST be named. Required.
`replace` string	Value to replace the captured group with.

SigV4Config

(Appears on:RemoteWriteSpec) SigV4Config specifies configuration to perform SigV4 authentication.

Fields

Field	Description
`region` string	Region of the AWS endpoint. If blank, the region from the default credentials chain is used.
`accessKey` Kubernetes core/v1.SecretKeySelector	AccessKey holds the secret of the AWS API access key to use for signing. If not provided, the environment variable AWS_ACCESS_KEY_ID is used.
`secretKey` Kubernetes core/v1.SecretKeySelector	SecretKey of the AWS API to use for signing. If blank, the environment variable AWS_SECRET_ACCESS_KEY is used.
`profile` string	Profile is the named AWS profile to use for authentication.
`roleARN` string	RoleARN is the AWS Role ARN to use for authentication, as an alternative for using the AWS API keys.

TemplateStageSpec

(Appears on:PipelineStageSpec) TemplateStageSpec is a transform stage that manipulates the values in the extracted map using Go's template syntax.

Fields

Field	Description
`source` string	Name from extracted data to parse. Required. If empty, defaults to using the log message.
`template` string	Go template string to use. Required. In addition to normal template functions, ToLower, ToUpper, Replace, Trim, TrimLeft, TrimRight, TrimPrefix, and TrimSpace are also available.

TenantStageSpec

(Appears on:PipelineStageSpec) TenantStageSpec is an action stage that sets the tenant ID for the log entry picking it from a field in the extracted data map.

Fields

Field	Description
`label` string	Name from labels whose value should be set as tenant ID. Mutually exclusive with source and value.
`source` string	Name from extracted data to use as the tenant ID. Mutually exclusive with label and value.
`value` string	Value to use for the template ID. Useful when this stage is used within a conditional pipeline such as match. Mutually exclusive with label and source.

TimestampStageSpec

(Appears on:PipelineStageSpec) TimestampStageSpec is an action stage that can change the timestamp of a log line before it is sent to Loki.

Fields

Field	Description
`source` string	Name from extracted data to use as the timestamp. Required.
`format` string	Determines format of the time string. Required. Can be one of: ANSIC, UnixDate, RubyDate, RFC822, RFC822Z, RFC850, RFC1123, RFC1123Z, RFC3339, RFC3339Nano, Unix, UnixMs, UnixUs, UnixNs.
`fallbackFormats` []string	Fallback formats to try if format fails.
`location` string	IANA Timezone Database string.
`actionOnFailure` string	Action to take when the timestamp can't be extracted or parsed. Can be skip or fudge. Defaults to fudge.

Release notes for Grafana Agent Operator

Wed, 11 Sep 2024 17:43:35 +0000

Release notes for Grafana Agent Operator

The release notes provide information about deprecations and breaking changes in Grafana Agent static mode Kubernetes operator.

For a complete list of changes to Grafana Agent, with links to pull requests and related issues when available, refer to the Changelog.

Note: These release notes are specific to the Static mode Kubernetes Operator. Other release notes for the different Grafana Agent variants are contained on separate pages:

Static mode release notes

Flow mode release notes

v0.33

Symbolic links in Docker containers removed

We’ve removed the deprecated symbolic links to /bin/agent* in Docker containers, as planned in v0.31. In case you’re setting a custom entrypoint, use the new binaries that are prefixed with /bin/grafana*.

v0.31

Breaking change: binary names are now prefixed with `grafana-`

As first announced in v0.29, the grafana-operator release binary names is now prefixed with grafana-:

agent-operator is now grafana-agent-operator.

For the grafana/agent-operator Docker container, the entrypoint is now /bin/grafana-agent-operator. A symbolic link from /bin/agent-operator to the new binary has been added.

Symbolic links will be removed in v0.33. Custom entrypoints must be updated prior to v0.33 to use the new binaries before the symbolic links get removed.

v0.29

Deprecation: binary names will be prefixed with `grafana-` in v0.31.0

The agent-operator binary name has been deprecated and will be renamed to grafana-agent-operator in the v0.31.0 release.

As part of this change, the Docker container for the v0.31.0 release will include symbolic links from the old binary names to the new binary names.

There is no action to take at this time.

v0.24

Breaking change: Grafana Agent Operator supported Agent versions

The v0.24.0 release of Grafana Agent Operator can no longer deploy versions of Grafana Agent prior to v0.24.0.

v0.19

Rename of Prometheus to Metrics (Breaking change)

As a part of the deprecation of “Prometheus,” all Operator CRDs and fields with “Prometheus” in the name have changed to “Metrics.”

This includes:

The PrometheusInstance CRD is now MetricsInstance (referenced by metricsinstances and not metrics-instances within ClusterRoles).
The Prometheus field of the GrafanaAgent resource is now Metrics
PrometheusExternalLabelName is now MetricsExternalLabelName

This is a hard breaking change, and all fields must change accordingly for the operator to continue working.

Note that old CRDs with the old hyphenated names must be deleted (kubectl delete crds/{grafana-agents,prometheus-instances}) for ClusterRoles to work correctly.

To do a zero-downtime upgrade of the Operator when there is a breaking change, refer to the new agentctl operator-detatch command: this will iterate through all of your objects and remove any OwnerReferences to a CRD, allowing you to delete your Operator CRDs or CRs.

Rename of CRD paths (Breaking change)

prometheus-instances and grafana-agents have been renamed to metricsinstances and grafanaagents respectively. This is to remain consistent with how Kubernetes names multi-word objects.

As a result, you will need to update your ClusterRoles to change the path of resources.

Example old ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-agent-operator
rules:
- apiGroups: [monitoring.grafana.com]
  resources:
  - grafana-agents
  - prometheus-instances
  verbs: [get, list, watch]

Example new ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-agent-operator
rules:
- apiGroups: [monitoring.grafana.com]
  resources:
  - grafanaagents
  - metricsinstances
  verbs: [get, list, watch]