Kubernetes metricsConfiguring remote_write with Helm and kube-prometheus-stack

Configuring remote_write with Helm and kube-prometheus-stack

In this guide you’ll learn how to configure Prometheus’s remote_write feature to ship cluster metrics to Grafana Cloud.

This guide assumes you have installed kube-prometheus-stack in your Kubernetes cluster using the Helm package manager. To learn how to install Helm on your local machine, please see Install Helm from the Helm documentation. To learn how to install kube-prometheus-stack, please see Install Chart from the kube-prometheus-stack GitHub repo.

The kube-prometheus-stack Helm chart installs the kube-prometheus stack. The kube-prometheus stack configures Prometheus Operator with a default Prometheus-Alertmanager-Grafana stack and sets up preconfigured Grafana dashboards and Alertmanager alerts. It also configures a set of Prometheus scrape targets and sets up node-exporter and kube-state-metrics. Prometheus Operator is a sub-component of the kube-prometheus stack.

Prometheus Operator implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and controller code that abstract away the management and implementation details of running a given service on Kubernetes. To learn more about Kubernetes Operators, please see Operator pattern from the Kubernetes docs.

The Prometheus Operator provides a set of Kubernetes Custom Resources that simplify Prometheus, Grafana, and Alertmanagemer deployment and configuration. For example, using the ServiceMonitor Custom Resource, you can configure how groups of Kubernetes services should be monitored in YAML manifests. The Operator controller will then communicate with the K8s API server to monitor Service endpoints and automatically generate the required Prometheus scrape configurations for the configured Services. To learn more about Prometheus Operator, please see the Prometheus Operator GitHub repository.

If you did not use Helm to install kube-prometheus please see Configuring remote_write with Prometheus Operator.

Step 1 — Create a Kubernetes Secret to store Grafana Cloud credentials

Begin by creating a Kubernetes Secret to store your Grafana Cloud Metrics username and password.

You can find your username by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.

Your password corresponds to an API key that you can generate by clicking on Generate now in this same panel. To learn how to create a Grafana Cloud API key, please see Create a Grafana Cloud API key.

Once you’ve noted your Cloud Prometheus username and password, create the Kubernetes secret. You can create a Secret by using a manifest file or create it directly using kubectl. In this guide we’ll create it directly using kubectl. To learn more about Kubernetes Secrets, please consult Secrets from the Kubernetes docs.

Run the following command to create a Secret called kubepromsecret:

kubectl create secret generic kubepromsecret \
  --from-literal=username=<your_grafana_cloud_prometheus_username>\
  --from-literal=password='<your_grafana_cloud_API_key>'\
  -n default

If you deployed your monitoring stack in a namespace other than default, change the -n monitoring flag to the appropriate namespace in the above command. To learn more about this command, please see Managing Secret using kubectl from the official Kubernetes docs.

Now that you’ve created a Secret to store your Grafana Cloud credentials, you can move on to modifying Prometheus’s configuration using a Helm values file.

Step 2 — Create a Helm values file with Prometheus remote_write configuration

In this step we’ll create a Helm values file to define parameters for Prometheus’s remote_write configuration. A Helm values file allows you to set configuration variables that are passed in to Helm’s object templates. To see the default values file for kube-prometheus-stack, consult values.yaml from the kube-prometheus-stack GitHub repository.

We’ll first create a values.yaml file defining Prometheus’s remote_write configuration, and then apply the new configuration to kube-prometheus-stack.

Open a file named new_values.yaml in your favorite editor. Paste in the following values:

prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "<Your Metrics instance remote_write endpoint>"
      basicAuth:
          username:
            name: kubepromsecret
            key: username
          password:
            name: kubepromsecret
            key: password

Here we set the remote_write URL and basic_auth username and password using the Secret created in the previous step.

When you’re done editing the file, save and close it.

Roll out the changes using helm upgrade -f:

helm upgrade -f new_values.yaml [your_release_name] prometheus-community/kube-prometheus-stack

Replace [your_release_name] with the name of the release you used to install kube-prometheus-stack. You can get a list of installed releases using helm list.

After running helm upgrade, you should see the following output:

Release "your_release_name" has been upgraded. Happy Helming!
NAME: your_release_name
LAST DEPLOYED: Mon Dec  7 17:29:03 2020
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace default get pods -l "release=your_release_name"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

At this point, you’ve successfully configured your Prometheus instances to remote_write scraped metrics to Grafana Cloud. You can verify that your changes have propagated to your running Prometheus instances using port-forward:

kubectl --namespace default port-forward svc/<your_release_name>-kube-prometheus-sta-prometheus 9090

Replace namespace with the appropriate namespace, and <your_release_name> with the Helm release name.

Navigate to http://localhost:9090 in your browser, and then Status and Configuration. Verify that the remote_write block you appended above has propagated to your running Prometheus instances.

Finally, log in to your Grafana instance to begin querying your cluster data. You can use the Billing/Usage dashboard to inspect incoming data rates in the last 5 minutes to confirm the flow of data to Grafana Cloud.