This guide gives a high-level overview of how the Grafana Agent Operator works.
The Grafana Agent Operator works in two phases:
- Discover a hierarchy of custom resources
- Reconcile that hierarchy into a Grafana Agent deployment
Custom Resource Hierarchy
The root of the custom resource hierarchy is the
GrafanaAgent resource. It is
primary resource the Operator looks for, and is called the “root” because it
discovers many other sub-resources.
The full hierarchy of custom resources is as follows:
Most of the resources above have the ability to reference a ConfigMap or a Secret. All referenced ConfigMaps or Secrets are added into the resource hierarchy.
When a hierarchy is established, each item is watched for changes. Any changed item will cause a reconcile of the root GrafanaAgent resource, either creating, modifying, or deleting the corresponding Grafana Agent deployment.
A single resource can belong to multiple hierarchies. For example, if two GrafanaAgents use the same Probe, modifying that Probe will cause both GrafanaAgents to be reconciled.
Build the Hierarchy
Grafana Agent Operator builds the hierarchy using label matching on the custom resources. The following figure illustrates the matching. The
GrafanaAgent picks up the
LogsInstance that match the label
instance: primary. The instances pick up the resources the same way.
Debug the Hierarchy
The generated configurations are saved in secrets. To download and validate them manually, use the following commands:
$ kubectl get secrets <???>-logs-config -o json | jq -r '.data."agent.yml"' | base64 --decode $ kubectl get secrets <???>-config -o json | jq -r '.data."agent.yml"' | base64 --decode
When a resource hierarchy is created, updated, or deleted, a reconcile occurs. When a GrafanaAgent resource is deleted, the corresponding Grafana Agent deployment will also be deleted.
Reconciling creates a few cluster resources:
- A Secret is generated holding the configuration of the Grafana Agent.
- Another Secret is created holding all referenced Secrets or ConfigMaps from the resource hierarchy. This ensures that Secrets referenced from a custom resource in another namespace can still be read.
- A Service is created to govern the created StatefulSets.
- One StatefulSet per Prometheus shard is created.
PodMonitors, Probes, and ServiceMonitors are turned into individual scrape jobs which all use Kubernetes SD.
Sharding and replication
The GrafanaAgent resource can specify a number of shards. Each shard results in the creation of a StatefulSet with a hashmod + keep relabel_config per job:
- source_labels: [__address__] target_label: __tmp_hash modulus: NUM_SHARDS action: hashmod - source_labels: [__tmp_hash] regex: CURRENT_STATEFULSET_SHARD action: keep
This allows for some decent horizontal scaling capabilities, where each shard will handle roughly 1/N of the total scrape load. Note that this does not use consistent hashing, which means changing the number of shards will cause anywhere between 1/N to N targets to reshuffle.
The sharding mechanism is borrowed from the Prometheus Operator.
The number of replicas can be defined, similarly to the number of shards. This creates duplicate shards. This must be paired with a remote_write system that can perform HA duplication. Grafana Cloud and Cortex provide this out of the box, and the Grafana Agent Operator defaults support these two systems.
The total number of created metrics pods will be product of
numShards * numReplicas.
Two labels are added by default to every metric:
cluster, representing the
GrafanaAgentdeployment. Holds the value of
__replica__, representing the replica number of the Agent. This label works out of the box with Grafana Cloud and Cortex’s HA deduplication.
The shard number is not added as a label, as sharding is designed to be transparent on the receiver end.