Connecting Prometheus-Ksonnet to Grafana Cloud

Published: 24 Feb 2020 RSS

In a previous post we showed how to install Prometheus and Grafana using the prometheus-ksonnet library along with Tanka. This is great for getting a well-managed monitoring install going, but sometimes it isn’t enough for monitoring larger clusters.

If you have multiple clusters that you want to monitor on a single dashboard, or need long-term storage, or need a high-availability setup for your monitoring data, then this installation won’t be sufficient on its own.

Fortunately, the Grafana Cloud Hosted Prometheus product offered by Grafana Labs solves all of these problems: It supports long-term storage, can receive metrics from multiple clusters, and is fundamentally designed to support high availability.

And it’s simple to connect Prometheus-Ksonnet to Grafana Cloud. This blog post will show you how.

Step-by-step instructions

  1. Log into or create an account on grafana.com.

  2. Click “Hosted Metrics.”

  3. Click “Create Hosted Metrics Instance.”

  4. Choose a name, select backend type Prometheus, then click “Create Hosted Metrics Instance.”

  5. You will now need to note a few configuration values that we will reuse later.
    Under “Grafana Data Source settings”:
    *Grab the URL. It will end with /api/prom.
    *Take note of your user ID (which will be a number).
    *Where it says password, click “Generate now.”

  6. Give this key a name (e.g. prometheus-ksonnet), choose a role of EditorViewer, and click “Create API Key.”

  7. You will be shown an API key. Make note of this. (Don’t worry; the keys shown in the screenshots are not valid.)

  8. Under the section “Sending Data to Prometheus,” you will see a snippet of YAML. Grab the URL from that YAML. It will end /api/prom/push.

  9. Now edit your environments/default/main.jsonnet.

    It should look like this, but with your own URLs, user ID, and api key in place of these samples:

    local prometheus = import 'prometheus-ksonnet/prometheus-ksonnet.libsonnet';
    
    prometheus {
    _config+:: {
    namespace: "default",
    cluster_name: "grafana",
    prometheus: {
      cluster: 'my-cluster',
      read_url: 'https://prometheus-us-central1.grafana.net/api/prom',
      write_url: 'https://prometheus-us-central1.grafana.net/api/prom/push',
      user: '10449',
      key: 'eyJrIjoiZTc2MzViYjk5MWQ4ZjgyN2Y4NDQwYTVhMGQ4OTZlZjhhNjUxZjJkOCIsIm4iOiJwcm9tZXRoZXVzLWtzb25uZXQiLCJpZCI6MTgxMDcxfQ==',
    },
    },
    prometheus_config+:: {
    global: { external_labels: {cluster: $._config.prometheus.cluster} },
    remote_write+: [
      {
        url: $._config.prometheus.write_url,
        basic_auth: {
          username: $._config.prometheus.user,
          password: $._config.prometheus.key,
        },
      },
    ],
    },
    grafana_datasource_config_map+:
    $.grafana_add_datasource_with_basicauth(
      'grafana-cloud', 
      $._config.prometheus.read_url, 
      $._config.prometheus.user, 
      $._config.prometheus.key),
    }
    

Here, you will see we have added values to configuration. We have added a remote write configuration to Prometheus so that it can find and write to Grafana Cloud, and we have added a data source to our Grafana instance. As well as this, we have added an external label to our Prometheus metrics that names our cluster. This means that we can distinguish metrics coming from separate clusters. We can use a template on our dashboards to allow us to change between clusters.

With this Grafana instance, we can now query our local Prometheus within our cluster, and we can also query the Grafana Cloud Hosted Prometheus instance. If this contains data regarding other clusters, we can access that data from any of our Grafana instances.

Interested in learning more?

To find out if Grafana Cloud – the highly available, fast, fully managed logging and metrics platform – is right for your organization, read more about it here.

Ready to try it now?

Related Posts

Check out this how-to for setting up monitoring in your Kubernetes cluster with Tanka and the Prometheus-Ksonnet library.
How The Trade Desk evolved their homegrown stack into a modern-day system that saves time and money
We're hosting our first-ever user group meetup in San Francisco. Sign up now!

Related Case Studies

DigitalOcean gains new insight with Grafana visualizations

The company relies on Grafana to be the consolidated data visualization and dashboard solution for sharing data.

"Grafana produces beautiful graphs we can send to our customers, works with our Chef deployment process, and is all hosted in-house."
– David Byrd, Product Manager, DigitalOcean

How Gojek is leveraging Cortex to keep up with its ever-growing scale

Gojek’s Lens monitoring system has 40+ tenants, for which Cortex handles about 1.2 million samples per second.

"The goal is to make sure that whenever a new service or team is created, they automatically get onboarded to the monitoring platform."
– Ankit Goel, Product Engineer, Gojek

How Grafana Cloud is enabling HotSchedules to develop next-generation applications

The visibility for all these metrics helps service delivery teams quickly iterate on new features.

"Grafana Cloud enables us to achieve observability bliss at HotSchedules. We don’t have to worry about scaling and maintaining the service."
– Denise Stockman, Director, Infrastructure, Hotschedules