Prometheus rules with cortextool

AlertsPrometheus rules with cortextool

This page outlines the steps to use the Grafana Cloud Prometheus-style rules during its early access phase. You can load Prometheus alerting and recording rules that are evaluated entirely in Grafana Cloud. This allows for global rule evaluation over all of the metrics stored in your Grafana Cloud stack.

Prometheus-style alerting is driven by both your Grafana Cloud Metrics and Grafana Cloud Alerts instances. The Metrics instance holds the recording and alerting rules definition, while the Alerts instance is in charge of routing and managing the alerts that fire from the Metrics instance. These are separate systems that must be individually configured in order for Prometheus alerting to work correctly.

The following sections cover both of these concepts:

  • How to upload recording and alerting rules definition to your Grafana Cloud Metrics instance
  • How to configure an Alertmanager for your Grafana Cloud Alerts instance, giving you access to the Alertmanager UI.

Note: You need an API key with proper permissions. You can use the same API key for both of your Metric and Alerting instances. Refer to Create a Grafana Cloud API key for instructions.

Download and install cortextool

cortextool is a powerful command-line tool for interacting with Cortex, which powers Grafana Cloud Metrics and Alerts. You’ll use cortextool to upload both, your recording / alerting rules definition and the Alertmanager configuration using YAML files.

Download or update to the latest version of cortextool, either as a binary or a Docker image:

If you use Docker, then enter docker run <docker image> -- <cortextool command>.

If you prefer to use the binary then follow the instructions published as part of each release.

You can find a list of the configurations supported by cortextool as parts of its README.

Note: For cortextool to interact with Grafana Cloud, you must set the correct configuration variables. Set them using either environment variables or a command line flags.

Upload recording and alerting rules definition to your Grafana Cloud Metrics instance

First, you’ll need to upload your recording and alerting rules to your Metrics instance. You’ll need the instance ID and the URL. These should be part of https://grafana.com/orgs/<yourOrgName>/.

Typically, your Metrics instance is likely to be in the us-central1 region. Its address would be in the form of https://prometheus-us-central1.grafana.net.

Using cortextool

With your instance ID, URL, and API key you’re now ready to upload your rules to your metrics instance. Use the following commands and files as a reference.

Example alert and rule definition YAML file. Take note of the namespace key which replaces the concept of “files” in this context given each instance only supports 1 configuration file.

# first_rules.yml
namespace: 'first_rules'
groups:
  - name: 'shopping_service_rules_and_alerts'
    rules:
      - alert: 'PromScrapeFailed'
        annotations:
          message: "Prometheus failed to scrape a target {{ $labels.job }}  / {{ $labels.instance }}"
        expr: "up != 1"
        for: "1m"
        labels:
          "severity": "critical"
      - record: "job:up:sum"
        expr: "sum by(job) (up)"

Although both recording and alerting rules are defined under the key rules the difference between a rule and and alert is generally (as there are others) whenever the key record or alert is defined.

With this file, you can run the following commands to upload your rules file in your Metrics instance. Keep in mind that these are example commands, they use placeholders and command line flags. The examples assume that files are located in the same directory.

$ cortextool rules load first_rules.yml \ 
--address=https://prometheus-us-central1.grafana.net \ 
--id=<yourID> \ 
--key=<yourKey>

Next, confirm that the rules were uploaded correctly by running:

$ cortextool rules list \ 
--address=https://prometheus-us-central1.grafana.net \ 
--id=<yourID> \ 
--key=<yourKey> 

Output is a list that shows you all the namespaces and rule groups for your instance ID:

Namespace   | Rule Group 
first_rules | shopping_service_rules_and_alerts 

You can also print the rules:

$ cortextool rules print \ 
--address=https://prometheus-us-central1.grafana.net \ 
--id=<yourID> \ 
--key=<yourKey> 

Output from the print command should look like this:

first_rules:
- name: shopping_service_rules_and_alerts
  interval: 0s
  rules:
    alert: PromScrapeFailed
    expr: up != 1
    for: 1m
    labels:
      severity: critical
    annotations:
      message: Prometheus failed to scrape a target {{ $labels.job }}  / {{ $labels.instance }}
  - record: job:up:sum
    expr: sum by(job) (up)

Upload Alertmanager configuration to your Grafana Cloud Alerts instance

To receive alerts you’ll need to upload your Alertmanager configuration to your Grafana Cloud Alerts instance. Similar to the previous step, you’ll need the corresponding instance ID, URL and API key. These should be part of https://grafana.com/orgs/<yourOrgName>/.

Typically, your Alerts instance is likely to be in the us-central1 region. Its address would be in the form of https://alertmanager-us-central1.grafana.net.

Using cortextool

With your instance ID, URL, and API key you’re now ready to upload your Alertmanager configuration to your Alerts instance. Use the following commands and files as a reference.

Ultimately, you’ll need to write your own or adapt an example config file for alerts to be delivered.

Example Alertmanager configuration. Please take that this not a working configuration, your alerts won’t be delivered with the following configuration but your Alertmanager UI will be accessible.

# alertmanager.yml
global: 
  smtp_smarthost: 'localhost:25' 
  smtp_from: 'youraddress@example.org' 
route: 
  receiver: example-email 
receivers: 
 - name: example-email 
   email_configs: 
    - to: 'youraddress@example.org' 

With this file, you can run the following commands to upload your Alertmanager configuration in your Alerts instance.

$ cortextool alertmanager load alertmanager.yml \ 
--address=https://alertmanager-us-central1.grafana.net \ 
--id=<yourID> \ 
--key=<yourKey> 

Then, confirm that the rules were uploaded correctly by running:

$ cortextool alertmanager get \ 
--address=https://alertmanager-us-central1.grafana.net \ 
--id=<yourID> \ 
--key=<yourKey> 

You should see output similar to the following:

global: 
  smtp_smarthost: 'localhost:25' 
  smtp_from: 'youraddress@example.org' 
route: 
  receiver: example-email 
receivers: 
 - name: example-email 
   email_configs: 
    - to: 'youraddress@example.org' 

Finally, you can delete the configuration with:

$ cortextool alertmanager delete \ 
--address=https://alertmanager-us-central1.grafana.net \ 
--id=<yourID> \ 
--key=<yourKey> 

UI access

After you upload a working Alertmanager configuration file, you can access the Alertmanager UI at: https://alertmanager-us-central1.grafana.net/alertmanager.