rclone integration for Grafana Cloud

Rclone is a command line program to manage files on Cloud storage. It is a feature-rich alternative to Cloud vendors’ web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business and consumer file storage services, and standard transfer protocols.

Use the walkthrough in Grafana Cloud to install the rclone integration.

Pre-installation configuration for the rclone integration

In order for rclone to emit metrics you must run it with the --rc-enable-metrics. This will also require either --rc-no-auth to disable authentication, or an --rc-user and --rc-pass to enable basic authentication for the endpoint.

For example, to run rclone with the remote control endpoint, enabling metrics, and with basic authentication you would run;

rclone rcd --rc-user <user> --rc-pass <rclone123> --rc-web-gui --rc-web-gui-no-open-browser --rc-enable-metrics --rc-no-auth --rc-addr :5572

Post-install configuration for the rclone integration

In the agent configuration file, the agent must be configured to scrape either a long running instance of rclone by hostname IP (in the example below rclone:5572). Alternatively, if you are using Kubernetes cron jobs, or otherwise doing batch synchronization, you must configure service discovery to find those ephemeral jobs.

An example of a static long running instance.

metrics:
  wal_directory: /tmp/wal
  configs:
    - name: integrations
      scrape_configs:
        - job_name: integrations/rclone
          static_configs:
            - targets: ['rclone:5572']
      remote_write:
        - url: http://cortex:9009/api/prom/push

An example of kubernetes service discovery which will find each execution. This is optional if you have a more broad kubernetes_sd_config which collects metrics for all pods. This example also requires you to provide basic authentication details for rclone.

metrics:
  wal_directory: /tmp/wal
  configs:
    - name: integrations
      scrape_configs:
        - job_name: integrations/rclone-kubernetes-pods
            basic_auth:
              username: <username>
              password: <password>
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - action: keep
                regex: rclone-job
                source_labels: [__meta_kubernetes_pod_label_name]
              - action: keep
                regex: .*-metrics
                source_labels:
                  - __meta_kubernetes_pod_container_port_name
              - action: replace
                regex: (https?)
                replacement: $1
                source_labels:
                  - __meta_kubernetes_pod_annotation_prometheus_io_scheme
                target_label: __scheme__
              - action: replace
                regex: (.+)
                replacement: $1
                source_labels:
                  - __meta_kubernetes_pod_annotation_prometheus_io_path
                target_label: __metrics_path__
              - action: replace
                regex: (.+?)(\:\d+)?;(\d+)
                replacement: $1:$3
                source_labels:
                  - __address__
                  - __meta_kubernetes_pod_annotation_prometheus_io_port
                target_label: __address__
              - action: replace
                replacement: $1
                separator: /
                source_labels:
                  - __meta_kubernetes_namespace
                  - __meta_kubernetes_pod_label_name
                target_label: job
              - action: replace
                source_labels:
                  - __meta_kubernetes_namespace
                target_label: namespace
              - action: replace
                source_labels:
                  - __meta_kubernetes_pod_name
                target_label: pod
              - action: replace
                source_labels:
                  - __meta_kubernetes_pod_container_name
                target_label: container
              - action: replace
                separator: ':'
                source_labels:
                  - __meta_kubernetes_pod_name
                  - __meta_kubernetes_pod_container_name
                  - __meta_kubernetes_pod_container_port_name
                target_label: instance
              - action: labelmap
                regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
                replacement: __param_$1
      remote_write:
        - url: http://cortex:9009/api/prom/push