Manage an instanceSetting per-instance resource usage limits

Overview

Per-instance resource limits can be used to ensure a single tenant cannot monopolize the resources of or threaten the stability of a Grafana Enterprise Metrics cluster. There are limits available to tune usage of all parts of the metric read and write paths. For a complete list of all available limits, refer to the Cortex limits configuration .

Using the admin API of Grafana Enterprise Metrics, you can easily set per-instance limits.

Setting limits via the admin API

Limits can be set using the admin API when creating a new instance or when modifying an existing instance.

Create a JSON payload, test1.json, for creating a new instance that includes a subset of per-instance limits.

{
    "name": "test1",
    "display_name": "Test Instance 1",
    "status": "active",
    "cluster": "metrics-enterprise-dev",
    "limits": {
        "max_series_per_query": 42000,
        "max_series_per_user": 42000
    }
}

Use curl to POST the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances \
    -X POST --data @test1.json

Read the newly created instance.

curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/test1 | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "metrics-enterprise-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_rate_strategy": "local",
    "ingestion_burst_size": 350000,
    "accept_ha_samples": false,
    "ha_cluster_label": "cluster",
    "ha_replica_label": "__replica__",
    "ha_max_clusters": 0,
    "drop_labels": null,
    "max_label_name_length": 1024,
    "max_label_value_length": 2048,
    "max_label_names_per_series": 30,
    "max_metadata_length": 1024,
    "reject_old_samples": false,
    "reject_old_samples_max_age": 1209600000000000,
    "creation_grace_period": 600000000000,
    "enforce_metadata_metric_name": true,
    "enforce_metric_name": true,
    "ingestion_tenant_shard_size": 0,
    "max_series_per_query": 42000,
    "max_samples_per_query": 100000,
    "max_series_per_user": 42000,
    "max_series_per_metric": 0,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "min_chunk_length": 0,
    "max_metadata_per_user": 8000,
    "max_metadata_per_metric": 10,
    "max_global_metadata_per_user": 0,
    "max_global_metadata_per_metric": 0,
    "max_chunks_per_query": 2000000,
    "max_query_lookback": "0s",
    "max_query_length": 0,
    "max_query_parallelism": 14,
    "cardinality_limit": 100000,
    "max_cache_freshness": 60000000000,
    "max_queriers_per_tenant": 0,
    "ruler_evaluation_delay_duration": 0,
    "ruler_tenant_shard_size": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0,
    "store_gateway_tenant_shard_size": 0,
    "compactor_blocks_retention_period": 0,
    "s3_sse_type": "",
    "s3_sse_kms_key_id": "",
    "s3_sse_kms_encryption_context": "",
    "per_tenant_override_config": "",
    "per_tenant_override_period": 10000000000
  }
}

Note that all available limit settings are populated, even ones that we did not include in our POST request. Any limit settings not included when a new instance is created with some limit settings are populated from the global default limits. This also applies when limits are added to an instance that previously did not have them: any limit settings that are not included will be populated from the global default limits.

Removing limits via the admin API

Limits can be removed from an instance using the admin API by modifying the instance.

Create a JSON payload, test1-update.json, that doesn’t include the limits field.

{
    "display_name": "Test Instance 1",
    "status": "active",
    "cluster": "metrics-enterprise-dev"
}

Use curl to PUT the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/test1 \
    -X PUT --data @test1-update.json | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "metrics-enterprise-dev"
}

The modified instance object is included in the update response.

Limit sources

Limits applied to an instance are only taken from a single location at a time:

These locations are checked for limits to apply to operations, in that order. Once a location for limits to apply to an operation is found, no other locations are checked. The effect of this is that if you set limits for an instance using the admin API, any limit settings for that instance in a runtime configuration file will be ignored.

Implementation

Enforcing limits based on values set with the admin API requires periodically making requests to object storage (S3, GCS, etc.) to get the most up to date limits for each instance. This means that based on the period you have configured to refresh instance-based limits (1 minute by default), N GET requests and 1 LIST request will be made to object storage, where N is the number of instances in your Grafana Enterprise Metrics cluster.