Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.
Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Add custom scrape jobs
Sometimes you want to add a scrape job for something that isn’t supported by the standard set of Prometheus Operator CRDs. A common example of this is node-level metrics.
To do this, you’ll need to write custom scrape configs and store it in a Kubernetes Secret:
apiVersion: v1
kind: Secret
metadata:
name: extra-jobs
namespace: operator
stringData:
jobs.yaml: |
<SCRAPE CONFIGS>
Replace <SCRAPE CONFIGS>
above with the array of Prometheus scrape jobs to
include.
For example, to collect metrics from Kubelet and cAdvisor, use the following:
apiVersion: v1
kind: Secret
metadata:
name: extra-jobs
namespace: operator
stringData:
jobs.yaml: |
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: integrations/kubernetes/kubelet
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
source_labels: [__meta_kubernetes_node_name]
replacement: /api/v1/nodes/$1/proxy/metrics
target_label: __metrics_path__
- action: hashmod
modulus: $(SHARDS)
source_labels:
- __address__
target_label: __tmp_hash
- action: keep
regex: $(SHARD)
source_labels:
- __tmp_hash
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: integrations/kubernetes/cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
- action: hashmod
modulus: $(SHARDS)
source_labels:
- __address__
target_label: __tmp_hash
- action: keep
regex: $(SHARD)
source_labels:
- __tmp_hash
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Note that you should always add these two relabel_configs for each custom job:
- action: hashmod
modulus: $(SHARDS)
source_labels:
- __address__
target_label: __tmp_hash
- action: keep
regex: $(SHARD)
source_labels:
- __tmp_hash
These rules ensure if your GrafanaAgent has multiple metrics shards, only one pod per replica will collect metrics for each job.
Once your Secret is defined, you’ll then need to add a additionalScrapeConfigs
field to your MetricsInstance:
apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
labels:
name: grafana-agent
name: primary
namespace: operator
spec:
additionalScrapeConfigs:
name: extra-jobs
key: jobs.yaml
# ... Other settings ...
The Secret MUST be in the same namespace as the MetricsInstance.