Use mimirtool to manage alerting resources
Use mimirtool
to create and upload alert and recording rules to your Grafana Cloud instance.
Once created, you can view these alert and recordiing rules from within the Grafana Cloud Alerting page in the UI.
Note
mimirtool
does not support Loki. For more information on using Grafana Alerting with Loki, refer to Alerting and recording rules for Loki.
Download and install mimirtool
mimirtool
is a powerful command-line tool for interacting with Grafana Mimir, which powers Grafana Cloud Metrics and Alerts. Use mimirtool
to upload your metric and log rules definition and the Alertmanager configuration using YAML files.
For more information, including installation instructions, see Grafana Mimirtool.
Note
To use
mimirtool
with Grafana Cloud, you need an API key with proper permissions. The same API key can be used for your metrics, log, and alerting instances.Additionally, set the configuration variables detailed in this guide, either as environment variables or command-line flags.
Upload rules definitions
Upload your alerting and recording rules to your Grafana Cloud Metrics and Logs instance. You need the instance ID and the URL. These should be part of /orgs/<yourOrgName>
/.
Metrics instance
Here is an example of a metrics instance address: https://prometheus-us-central1.grafana.net.
Logs instance
Here is an example of a logs instance address: https://logs-prod-us-central1.grafana.net.
With your instance ID, URL, and API key you’re now ready to upload your rules to your metrics instance. Use the following commands and files as a reference.
Below is an example alert and rule definition YAML file. Take note of the namespace key which replaces the concept of “files” in this context given each instance only supports 1 configuration file.
# first_rules.yml
namespace: 'first_rules'
groups:
- name: 'shopping_service_rules_and_alerts'
rules:
- alert: 'PromScrapeFailed'
annotations:
message: 'Prometheus failed to scrape a target {{ $labels.job }} / {{ $labels.instance }}'
expr: 'up != 1'
for: '1m'
labels:
'severity': 'critical'
- record: 'job:up:sum'
expr: 'sum by(job) (up)'
Although both recording and alerting rules are defined under the key rules
the difference between a rule and and alert is generally (as there are others) whenever the key record
or alert
is defined.
With this file, you can run the following commands to upload your rules file in your Metrics or Logs instance. Keep in mind that these are example commands for your Metrics instance, and they use placeholders and command line flags. Follow a similar pattern for your Logs instances by switching the address to the correct one. The examples also assume that files are located in the same directory.
$ mimirtool rules load first_rules.yml \
--address=https://prometheus-us-central1.grafana.net \
--id=<yourID> \
--key=<yourKey>
Next, confirm that the rules were uploaded correctly by running:
$ mimirtool rules list \
--address=https://prometheus-us-central1.grafana.net \
--id=<yourID> \
--key=<yourKey>
Output is a list that shows you all the namespaces and rule groups for your instance ID:
Namespace | Rule Group
first_rules | shopping_service_rules_and_alerts
You can also print the rules:
$ mimirtool rules print \
--address=https://prometheus-us-central1.grafana.net \
--id=<yourID> \
--key=<yourKey>
Output from the print command should look like this:
first_rules:
- name: shopping_service_rules_and_alerts
interval: 0s
rules:
- alert: PromScrapeFailed
expr: up != 1
for: 1m
labels:
severity: critical
annotations:
message: Prometheus failed to scrape a target {{ $labels.job }} / {{ $labels.instance }}
- record: job:up:sum
expr: sum by(job) (up)
Add an external Alertmanager
To receive alerts you need to upload your Alertmanager configuration to your Grafana Cloud Alerts instance. Similar to the previous step, you need the corresponding instance ID, URL and API key. These should be part of /orgs/<yourOrgName>
/.
Your Alerts instance is likely to be in the us-central1
region. Its address would be in the form of https://alertmanager-us-central1.grafana.net.
With your instance ID, URL, and API key you’re now ready to upload your Alertmanager configuration to your Alerts instance. Use the following commands and files as a reference.
Ultimately, you’ll need to write your own or adapt an example config file for alerts to be delivered.
Below is an example Alertmanager configuration. Please take that this not a working configuration, your alerts won’t be delivered with the following configuration but your Alertmanager UI will be accessible.
# alertmanager.yml
global:
smtp_smarthost: 'localhost:25'
smtp_from: 'youraddress@example.org'
route:
receiver: example-email
receivers:
- name: example-email
email_configs:
- to: 'youraddress@example.org'
With this file, you can run the following commands to upload your Alertmanager configuration in your Alerts instance.
$ mimirtool alertmanager load alertmanager.yml \
--address=https://alertmanager-us-central1.grafana.net \
--id=<yourID> \
--key=<yourKey>
Then, confirm that the rules were uploaded correctly by running:
$ mimirtool alertmanager get \
--address=https://alertmanager-us-central1.grafana.net \
--id=<yourID> \
--key=<yourKey>
You should see output similar to the following:
global:
smtp_smarthost: 'localhost:25'
smtp_from: 'youraddress@example.org'
route:
receiver: example-email
receivers:
- name: example-email
email_configs:
- to: 'youraddress@example.org'
Finally, you can delete the configuration with:
$ mimirtool alertmanager delete \
--address=https://alertmanager-us-central1.grafana.net \
--id=<yourID> \
--key=<yourKey>
After you upload a working Alertmanager configuration file, you can access the Alertmanager UI at: https://alertmanager-us-central1.grafana.net/alertmanager.