Grafana Kubernetes Monitoring
Other configuration methods
Use remote_write to ship Prometheus metrics
Configure remote_write with Helm and PrometheusConfigure remote_write with Helm and Prometheus
In this guide, you’ll learn how to configure Prometheus’s remote_write feature with Helm to ship cluster metrics to Grafana Cloud using Helm.
Before you start
This guide assumes you have installed the Prometheus Helm chart in your Kubernetes cluster using the Helm package manager. To learn how to install Helm on your local machine, see Install Helm from the Helm documentation. To learn how to install the Prometheus chart, see Install Chart from the Prometheus chart GitHub repo.
The Prometheus Helm chart installs and bootstraps a one-replica Prometheus Deployment into your Kubernetes cluster. It also sets up kube-state-metrics, Pushgateway, Alertmanager, and node-exporter. It additionally configures a default set of Kubernetes observability scraping jobs for Prometheus. The Prometheus Helm chart provides a more lightweight foundation to build from than kube-prometheus-stack and can be useful if you don’t want to use Prometheus Operator or run a local Grafana instance. To learn more, see the Prometheus Helm chart GitHub repo.
If you did not use Helm to install Prometheus into your cluster or are using Prometheus Operator and the kube-prometheus stack, select a different method.
Create a Helm values file containing the remote_write configuration
In this step we’ll create a Helm values file to define parameters for Prometheus’s remote_write configuration. A Helm values file allows you to set configuration variables that are passed in to Helm’s object templates. To see the default values file for the Prometheus Helm chart, consult values.yaml from the Prometheus Helm chart GitHub repository.
We’ll first create a values file defining Prometheus’s remote_write configuration, and then apply this new configuration to the Prometheus deployment running in our cluster.
Open a file named new_values.yaml in your favorite editor. Paste in the following values:
server:
remoteWrite:
- url: "<Your Metrics instance remote_write endpoint>"
basic_auth:
username: <your_grafana_cloud_prometheus_username>
password: <your_grafana_cloud_API_key>
Here we set the remote_write URL and basic_auth username and password using our Grafana Cloud credentials.
You can find your username by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.
Your password corresponds to an API key that you can generate by clicking on Generate now in this same panel. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key
When you’re done editing the file, save and close it.
Now that you’ve created a values file with your Prometheus remote_write configuration, you can move on to upgrading the Prometheus Helm chart.
Upgrade the Prometheus Helm chart
Upgrade the Prometheus Helm chart with the values file you just created using helm upgrade -f:
helm upgrade -f new_values.yml [your_release_name] prometheus-community/prometheus
Replace [your_release_name] with the name of the release you used to install Prometheus. You can get a list of installed releases using helm list.
After running helm upgrade, you should see the following output:
Release "[your_release_name]" has been upgraded. Happy Helming!
NAME: [your_release_name]
LAST DEPLOYED: Thu Dec 10 16:41:33 2020
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
[your_release_name]-prometheus-server.default.svc.cluster.local
. . .
Get the PushGateway URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9091
For more information on running Prometheus, visit:
https://prometheus.io/
At this point, you’ve successfully configured Prometheus to remote_write scraped metrics to Grafana Cloud. You can verify that your running Prometheus instance is remote_writing correctly using port-forward.
First, get the Prometheus server Service name:
kubectl get svc
The Prometheus Service name should look something like <your_release_name>-prometheus-server.
Next, use port-forward to forward a local port to the Prometheus Service:
kubectl --namespace default port-forward svc/<prometheus-service-name> 9090:80
Replace namespace with the appropriate namespace, and <prometheus-service-name> with the name of the Prometheus service.
Navigate to http://localhost:9090 in your browser, and then Status and Configuration. Verify that the remote_write block you created above has propagated to your running Prometheus instance configuration. It may take a couple of minutes for the changes to get picked up by the running Prometheus instance.
Finally, log in to your Grafana instance to begin querying your cluster data. You can use the Billing/Usage dashboard to inspect incoming data rates in the last 5 minutes to confirm the flow of data to Grafana Cloud.
