rclone integration for Grafana Cloud
Rclone is a command line program to manage files on Cloud storage. It is a feature-rich alternative to Cloud vendors’ web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business and consumer file storage services, and standard transfer protocols.
Use the walkthrough in Grafana Cloud to install the rclone integration.
Pre-installation configuration for the rclone integration
In order for rclone to emit metrics you must run it with the --rc-enable-metrics
. This will also require either --rc-no-auth
to disable authentication, or an --rc-user
and --rc-pass
to enable basic authentication for the endpoint.
For example, to run rclone with the remote control endpoint, enabling metrics, and with basic authentication you would run;
rclone rcd --rc-user <user> --rc-pass <rclone123> --rc-web-gui --rc-web-gui-no-open-browser --rc-enable-metrics --rc-no-auth --rc-addr :5572
Post-install configuration for the rclone integration
In the agent configuration file, the agent must be configured to scrape either a long running instance of rclone by hostname IP (in the example below rclone:5572
). Alternatively, if you are using Kubernetes cron jobs, or otherwise doing batch synchronization, you must configure service discovery to find those ephemeral jobs.
An example of a static long running instance.
metrics:
wal_directory: /tmp/wal
configs:
- name: integrations
scrape_configs:
- job_name: integrations/rclone
static_configs:
- targets: ['rclone:5572']
remote_write:
- url: http://cortex:9009/api/prom/push
An example of kubernetes service discovery which will find each execution. This is optional if you have a more broad kubernetes_sd_config
which collects metrics for all pods. This example also requires you to provide basic authentication details for rclone.
metrics:
wal_directory: /tmp/wal
configs:
- name: integrations
scrape_configs:
- job_name: integrations/rclone-kubernetes-pods
basic_auth:
username: <username>
password: <password>
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: rclone-job
source_labels: [__meta_kubernetes_pod_label_name]
- action: keep
regex: .*-metrics
source_labels:
- __meta_kubernetes_pod_container_port_name
- action: replace
regex: (https?)
replacement: $1
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
replacement: $1
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: (.+?)(\:\d+)?;(\d+)
replacement: $1:$3
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_label_name
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
separator: ':'
source_labels:
- __meta_kubernetes_pod_name
- __meta_kubernetes_pod_container_name
- __meta_kubernetes_pod_container_port_name
target_label: instance
- action: labelmap
regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
replacement: __param_$1
remote_write:
- url: http://cortex:9009/api/prom/push
Related Grafana Cloud resources
Intro to Prometheus and Grafana Cloud
Prometheus is taking over the monitoring world! In this webinar, we will start with a quick introduction to the open source project that’s the de facto standard for monitoring modern, cloud native systems.
How to set up and visualize synthetic monitoring at scale with Grafana Cloud
Learn how to use Kubernetes, Grafana Loki, and Grafana Cloud’s synthetic monitoring feature to set up your infrastructure's checks in this GrafanaCONline session.
Using Grafana Cloud to drive manufacturing plant efficiency
This GrafanaCONline session tells how Grafana helps a 75-year-old manufacturing company with product quality and equipment maintenance.