--- title: "Adaptive Metrics HTTP API | Grafana Cloud documentation" description: "Use the Adaptive Metrics HTTP API to manage your Adaptive Metrics configuration at scale and integrate it into your existing infrastructure-as-code workflows." --- # Adaptive Metrics HTTP API The following topic describes the Adaptive Metrics HTTP API. ### Before you begin To interact with the API, gather the following information: - `URL`: In the form `https://.grafana.net`. To find your `URL` value, go to your [`grafana.com`](/) account and check the **Details** page of your hosted Prometheus endpoint. - `TENANT`: The numeric instance ID where you want to use Adaptive Metrics. To find your `TENANT` value, go to your [grafana.com](/) account and check the **Details** page of your hosted Prometheus endpoint for **Username / Instance ID**. - `TOKEN`: A token from a Grafana Cloud Access Policy that has the required permissions to interact with the Adaptive Metrics resources for your stack ID, as shown in the following examples. More details about available permissions and scopes can be found in [RBAC for Adaptive Metrics](/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/additional-configuration/adaptive-metrics-rbac/). > Note > > You can create [Grafana Cloud Access Policies](../../../../security-and-account-management/authentication-and-permissions/access-policies/#use-with-the-api-grafana-stacks-and-cloud-portal) from the Grafana portal, with the API, or in your Grafana Cloud Stack. ### The format of an aggregation rule Each aggregation rule looks similar to this: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none { "metric": "agent_request_duration_seconds_sum", "drop_labels": [ "container", "instance", "method", "namespace", "pod", "provider", "status_code", "ws" ], "aggregations": [ "sum:counter" ] } ``` In the preceding example: - `metric` is the name of the metric to be aggregated. - `drop_labels` is an array of the labels that are aggregated and their value is set to ``. - `aggregations` is an array of the aggregation types to calculate for this metric. You can use an aggregation rule file to define multiple rules simultaneously. The following example rule file is an array of one or more aggregation rules: JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json [ { "metric": "agent_request_duration_seconds_sum", "drop_labels": ["namespace", "pod"], "aggregations": ["sum:counter"] }, { "metric": "prometheus_request_duration_seconds_sum", "drop_labels": ["container", "instance", "ws"], "aggregations": ["sum:counter"] } ] ``` ### List recommendations Download recommendations for metrics to aggregate using the command below. `TOKEN` and `TENANT` are variables defined within the [requirements section](#before-you-begin). Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations" ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-recommendations:read`** scope. You can use an optional verbose flag to retrieve more information about each recommendation: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations?verbose=true" ``` The following is example of verbose recommendation rule: JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json [ { "metric": "probe_http_duration_seconds", "drop_labels": ["config_version", "group", "method", "name", "url"], "aggregations": ["count", "sum"], "recommended_action": "add", "usages_in_rules": 0, "usages_in_queries": 0, "usages_in_dashboards": 2, "kept_labels": ["name", "instance", "job"], "total_series_after_aggregation": 1040, "total_series_before_aggregation": 1210 }, { "metric": "probe_dns_duration_seconds", "drop_labels": ["config_version"], "aggregations": ["count", "sum"], "recommended_action": "add", "usages_in_rules": 0, "usages_in_queries": 0, "usages_in_dashboards": 2, "kept_labels": ["name", "instance", "job"], "total_series_after_aggregation": 264, "total_series_before_aggregation": 396 } ] ``` Any existing aggregation rules that have been previously applied are included in the downloaded list of recommendations. This simplifies the process of viewing the recommended aggregation rules and applying them. By default, you can apply recommended rules and still maintain all of your existing rules. > Note > > Recommendations are generated asynchronously from the API calls; the API call only returns the most recently generated set of recommendations. ### List current recommendations configuration Download the current configuration of the recommendations service using the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations/config" ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-config:read`** scope. The only tunable parameter exposed by the recommendations service is the `keep_labels` parameter. This parameter allows the user to define a list of labels that they never want recommended for aggregation. This can be useful at organizations where certain labels are always expected on metrics, regardless of whether or not those labels have been recently queried. > Note > > The `keep_labels` parameter only applies to new recommendations; existing rules that contain labels specified in `keep_labels` remain unchanged. An example response from the `/recommendations/config` endpoint would look as follows: JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "keep_labels": ["instance", "pod"] } ``` The preceding response indicates that the recommendations service has been configured to never recommend aggregating the `instance` or `pod` labels. ### Recommendation exemptions The Adaptive Metrics API provides the possibility to exempt certain metrics, labels, and metric-label combinations from generated recommendations. The recommendation exemption format looks like this: Expand table | Field name | Type | Description | |--------------------------|--------------|------------------------------------------------------------------------------------------------------------------------------------------------| | `id` | identifier | Unique identifier of the exemption (set by the server). | | `metric` (optional) | metric name | Metric name or metric name matcher to be exempted from recommendations. | | `match_type` (optional) | string | The type of matching to apply to the `metric` field. Valid values are `exact`, `prefix`, and `suffix`. If not specified, the value is `exact`. | | `keep_labels` (optional) | string array | List of labels to not be considered for aggregation by the recommendation system. | > Note > > For an exemption to be considered valid, either the `metric` or `keep_labels` field must be defined. Below are examples of exemptions and how they would be reflected in the preceding mentioned format: 1. Completely exclude metric `api_http_requests_total` from being recommended for aggregation (keep full cardinality). JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "metric": "api_http_requests_total" } ``` 2. Exclude labels `namespace` and `status` for metric `api_http_requests_total` from being recommended for aggregation. JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "metric": "api_http_requests_total", "keep_labels": ["namespace", "status"] } ``` 3. Exclude the `pod` label for all metrics from being recommended for aggregation. JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "keep_labels": ["pod"] } ``` 4. Exclude all metrics beginning with `http_requests_` from being recommended for aggregation. JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "metric": "http_requests_", "match_type": "prefix" } ``` 5. Exclude all metrics ending with `_total` from being recommended for aggregation. JSON ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```json { "metric": "_total", "match_type": "suffix" } ``` #### Create a recommendation exemption Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" --request POST --data @exemption.json "$URL/v1/recommendations/exemptions ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-exemptions:write`** scope. If successful, the server returns a JSON object including the `id` of the newly created exemption. ##### List recommendation exemptions Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" "$URL/v1/recommendations/exemptions ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-exemptions:read`** scope. #### Get a recommendation exemption Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" "$URL/v1/recommendations/exemptions/ ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-exemptions:read`** scope. The `exemption_id` is the identifier of the exemption to be fetched, as returned by the server when creating the exemption. ##### Update recommendation exemption Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" --request PUT --data @exemption.json "$URL/v1/recommendations/exemptions/ ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-exemptions:write`** scope. The `exemption_id` is the identifier of the exemption to be updated, as returned by the server when creating the exemption. #### Delete a recommendation exemption Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" --request DELETE "$URL/v1/recommendations/exemptions/ ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-exemptions:write`** scope. The `exemption_id` is the identifier of the exemption to delete, as returned by the server when creating the exemption. ### Update recommendations configuration Upload new recommendations configuration using the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" --request POST --data @config.json "$URL/aggregations/recommendations/config" ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-recommendations:write`** scope. This command uses the same endpoint described in [List current recommendations configuration](#list-current-recommendations-configuration) and expects the same JSON format. ### List currently applied aggregation rules Download your existing aggregation rules: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" "$URL/aggregations/rules" ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-rules:read`** scope. ### Upload new aggregation rules Uploading new aggregation rules is a multi-step process: 1. Fetch the currently applied rules. 2. Modify rules locally. 3. Upload rules back. There is a delay between uploading new aggregation rules and those metrics aggregations taking effect in your environment. In most cases, the delay is approximately 5-10 minutes, but we currently have no mechanism to let you know precisely when new aggregations take effect. Grafana currently limits how often new aggregation rules are applied. Although you can upload as many new versions of your aggregation rules as you like, those updates are only applied once every 10 minutes. If you make multiple updates in quick succession, the system applies your first received (oldest) update. Then, 10 minutes later, the most recently received update is applied. The intermediate updates never get applied. #### Fetch the currently applied rules Use this command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash curl -u "$TENANT:$TOKEN" -D headers.txt "$URL/aggregations/rules" > rules.json ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-rules:read`** scope. The preceding command uses an additional `-D headers.txt` argument, which stores the headers in a file called `headers.txt.` This step is required if you want to then upload a new rule file, for example if you want to update the existing aggregation rules you have in place. The information in these headers ensures there are no *update collisions*. An update collision is the scenario where multiple users try to edit the rules file at the same time and overwrite one another’s changes. #### Modify the rules locally Use your editor of choice to modify the `rules.json` file downloaded in the prior step. #### Upload rules back The API supports uploading an entire rules file. > Warning > > This action overwrites your existing rule file. To upload your modified `rules.json` file from the previous step, use the following shell script: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash TMPFILE=$(mktemp) trap 'rm "$TMPFILE"' EXIT cat headers.txt | grep -i '^etag:' | sed 's/^ETag:/If-Match:/i' > "$TMPFILE" curl --request POST --header @"$TMPFILE" --data-binary @$1 -u "$TENANT:$TOKEN" "$URL/aggregations/rules" ``` `TOKEN` must belong to an access policy with the **`adaptive-metrics-rules:write`** scope. The `cat headers.txt`command modifies the `headers.txt` file created in the previous curl call that pulled down the existing aggregation rules. The `curl --request POST`command uploads your new rules file, as well as the updated headers. Save the shell script as `rules_upload.sh.` To run that script, use the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash ./rules_upload.sh ``` Replace `` with the name of the rules file you wish to upload. > Note > > If, upon trying to `POST` the new rules file, you see the error `the Etag supplied in the 'If-Match' header does not match the Etag of the rules you are trying to replace`, the headers you provided are either missing or stale. To fix, re-fetch the rules file and headers, being careful to look for any changes that may have been introduced since your last edits. For more information on Etag headers, see [Etag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag). > Note > > After you configure aggregation rules, the [active series](/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/metrics-invoice/#active-series) count might increase temporarily. Aggregated and non-aggregated series are considered active at the same time. After a short period of time, the non-aggregated series are no longer considered active, and you see a net reduction in active series.