This is archived documentation for v1.5.0. Go to the latest version.

Manage an instanceSetting per-instance resource usage limits

Overview

Per-instance resource limits can be used to ensure a single tenant cannot monopolize the resources of or threaten the stability of a Grafana Enterprise Metrics cluster. There are limits available to tune usage of all parts of the metric read and write paths. For a complete list of all available limits, refer to the Cortex limits configuration.

Using the admin API of Grafana Enterprise Metrics, you can easily set per-instance limits.

Setting limits via the admin API

Limits can be set using the admin API when creating a new instance or when modifying an existing instance.

Create a JSON payload, test1.json, for creating a new instance that includes a subset of per-instance limits.

{
    "name": "test1",
    "display_name": "Test Instance 1",
    "status": "active",
    "cluster": "metrics-enterprise-dev",
    "limits": {
        "max_series_per_query": 42000,
        "max_series_per_user": 42000
    }
}

Use curl to POST the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances \
    -X POST --data @test1.json

Read the newly created instance.

curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/test1 | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "metrics-enterprise-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_rate_strategy": "local",
    "ingestion_burst_size": 350000,
    "accept_ha_samples": false,
    "ha_cluster_label": "cluster",
    "ha_replica_label": "__replica__",
    "ha_max_clusters": 0,
    "drop_labels": null,
    "max_label_name_length": 1024,
    "max_label_value_length": 2048,
    "max_label_names_per_series": 30,
    "max_metadata_length": 1024,
    "reject_old_samples": false,
    "reject_old_samples_max_age": 1209600000000000,
    "creation_grace_period": 600000000000,
    "enforce_metadata_metric_name": true,
    "enforce_metric_name": true,
    "ingestion_tenant_shard_size": 0,
    "max_series_per_query": 42000,
    "max_samples_per_query": 100000,
    "max_series_per_user": 42000,
    "max_series_per_metric": 0,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "min_chunk_length": 0,
    "max_metadata_per_user": 8000,
    "max_metadata_per_metric": 10,
    "max_global_metadata_per_user": 0,
    "max_global_metadata_per_metric": 0,
    "max_chunks_per_query": 2000000,
    "max_query_lookback": "0s",
    "max_query_length": 0,
    "max_query_parallelism": 14,
    "cardinality_limit": 100000,
    "max_cache_freshness": 60000000000,
    "max_queriers_per_tenant": 0,
    "ruler_evaluation_delay_duration": 0,
    "ruler_tenant_shard_size": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0,
    "store_gateway_tenant_shard_size": 0,
    "compactor_blocks_retention_period": 0,
    "s3_sse_type": "",
    "s3_sse_kms_key_id": "",
    "s3_sse_kms_encryption_context": "",
    "per_tenant_override_config": "",
    "per_tenant_override_period": 10000000000
  }
}

Note that all available limit settings are populated, even ones that we did not include in our POST request. Any limit settings not included when a new instance is created with some limit settings are populated from the global default limits. This also applies when limits are added to an instance that previously did not have them: any limit settings that are not included will be populated from the global default limits.

Removing limits via the admin API

Limits can be removed from an instance using the admin API by modifying the instance.

Create a JSON payload, test1-update.json, that doesn’t include the limits field.

{
    "display_name": "Test Instance 1",
    "status": "active",
    "cluster": "metrics-enterprise-dev"
}

Use curl to PUT the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/test1 \
    -X PUT --data @test1-update.json | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "metrics-enterprise-dev"
}

The modified instance object is included in the update response.

Limit sources

Limits applied to an instance are only taken from a single location at a time:

These locations are checked for limits to apply to operations, in that order. Once a location for limits to apply to an operation is found, no other locations are checked. The effect of this is that if you set limits for an instance using the admin API, any limit settings for that instance in a runtime configuration file will be ignored.

Implementation

Enforcing limits based on values set with the admin API requires periodically making requests to object storage (S3, GCS, etc.) to get the most up to date limits for each instance. This means that based on the period you have configured to refresh instance-based limits (1 minute by default), N GET requests and 1 LIST request will be made to object storage, where N is the number of instances in your Grafana Enterprise Metrics cluster.

Setting per tenant retention of metrics

To limit the amount of time a tenant can store metrics set compactor_blocks_retention_period for that tenant. The setting is a duration. It accepts values like 2y meaning 2 years and 60d meaning 60 days. After being marked for deletion the blocks are deleted by the blocks cleaner after a time indicated by the deletion_delay configuration setting (default=12h) has passed.

Example

Create a JSON payload, retention.json, for creating a new instance that includes a retention period.

{
    "name": "retention",
    "display_name": "With retention",
    "status": "active",
    "cluster": "metrics-enterprise-dev",
    "limits": {
        "compactor_blocks_retention_period": "1y"
    }
}

Use curl to POST the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances \
    -X POST --data @retention.json