Analyzing metrics usage with the Prometheus API

If you have a large number of active series or larger endpoints (100k’s of series and bigger), the analytical Prometheus queries might run longer than the Grafana Explorer is configured to wait (for results to be available). In this case we recommend directly interacting with the Prometheus HTTP API. That’s what this page is all about.

Prerequisites

To begin with this section, you should have the jq, curl, sort, and date command-line utilities installed on your machine. You can also use an API request tool like Postman, but configuring such tools goes beyond the scope of this guide.

Open a command-line shell like bash and set the following two variables that will be referenced in subsequent curl commands:

login="your_endpoint_user:your_grafana_cloud_API_token>"
url="your_metrics_instance_query_endpoint"

Be sure to replace your_endpoint_user with the User parameter from the Prometheus endpoint Details page. You can navigate to the Prometheus endpoint details page from the Cloud Portal:

In the example above, the User is 18818.

Replace your_grafana_cloud_API_token with a Viewer role API key. You can create this by following the instructions in Create a Grafana Cloud API Key.

Once you’ve set the login and url variables, you can begin querying the Prometheus API.

Fetch a list of active metrics and their cardinalities

You can fetch a list of active metrics and cardinalities using the /query API endpoint, which accepts a PromQL query as a parameter.

We’ll first query the API for a list of metrics, and then loop over this list, querying the API once again for an active series count for each metric.

First, fix the now variable to the current time:

now=$(date +%s)

Then, run the loop using curl:

curl -s -u $login $url/api/v1/label/__name__/values \
| jq -r ".data[]" \
| while read metric; do
    count=$(curl -s \
        -u $login \
        --data-urlencode 'query=count({__name__="'$metric'"})' \
        --data-urlencode "time=$now" \
        $url/api/v1/query \
    | jq -r ".data.result[0].value[1]")
    echo "$count $metric"
done

This command first queries the /api/v1/label/ metadata endpoint with the __name__ meta-label to fetch a list of metric names. It then uses the jq utility to extract the data field from the JSON response.

Finally, the while loop iterates over the metrics, querying the API at /api/v1/query with the metric name and count() operator for a count of active series for each metric. The metric cardinality and metric name are then printed to stdout.

You can save the output to a file by appending > metric_cardinalities.out to the end of the command. You can then import this data into a spreadsheet for further analysis.

Fetch a list of active series for a given metric

To further drill down into a high cardinality metric, you can fetch a list of active series for any given metric. To do this, begin by once again fixing the now variable to the current time:

now=$(date +%s)

Set the metric variable to the metric you’d like to query:

metric=high_cardinality_metric

Be sure to replace high_cardinality_metric with the metric you’d like to query.

Then, use curl to perform the following request against the Prometheus API:

curl -s \
    -u $login \
    --data-urlencode "query=$metric" \
    --data-urlencode "time=$now" \
    $url/api/v1/query \
| jq -c ".data.result[].metric"

This command uses the Prometheus API’s Query endpoint to fetch a list of active time series for a given metric. To learn more about the Prometheus HTTP API, please consult the Prometheus API docs.

Fetch a list of all metrics

Note: This method may report some inactive metrics.

You can also use the metadata API to quickly fetch a list of all metrics.

To begin, ensure that you’ve set the login and url variables correctly as detailed in the Prerequisites.

Next, perform the following HTTP request using curl:

curl -s -u $login $url/api/v1/label/__name__/values | jq -r ".data[]" | sort

This queries the /api/v1/label endpoint for metric names using the __name__ meta-label. The result is then extracted using jq and sorted using the sort utility.

Fetch a list of labels and their cardinalities

Note: This method may report some inactive metrics.

To fetch a list of labels and their cardinalities (for example the number of unique hostname values), you can use the metadata API.

To begin, ensure that you’ve set the login and url variables correctly as detailed in the Prerequisites.

Now, perform the following HTTP request using curl:

curl -s -u $login $url/api/v1/labels \
| jq -r ".data[]" \
| while read label; do
    count=$(curl -s -u $login $url/api/v1/label/$label/values \
    | jq -r ".data|length")
    echo "$count $label"
  done \
| sort -n

This request fetches a list of label names and then loops through them, fetching a list of label values for each label name. jq counts the length of the response list and the output is printed to stdout. The sort utility sorts the resulting count label list by label cardinality, which helps you identify the highest cardinality labels (labels with the most values).