Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Grafana Enterprise Metrics DocumentationManage an instanceSetting per-instance resource usage limitsOverview
Per-instance resource limits can be used to ensure a single tenant cannot monopolize the resources of or threaten the stability of a Grafana Enterprise Metrics cluster. There are limits available to tune usage of all parts of the metric read and write paths. For a complete list of all available limits, refer to the Cortex limits configuration.
Using the admin API of Grafana Enterprise Metrics, you can easily set per-instance limits.
Setting limits via the admin API
Limits can be set using the admin API when creating a new instance or when modifying an existing instance.
Create a JSON payload, test1.json, for creating a new instance that includes a subset of per-instance limits.
{
"name": "test1",
"display_name": "Test Instance 1",
"status": "active",
"cluster": "metrics-enterprise-dev",
"limits": {
"max_series_per_query": 42000,
"max_series_per_user": 42000
}
}Use curl to POST the data to the admin API.
$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances \
-X POST --data @test1.jsonRead the newly created instance.
curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/test1 | jq
{
"name": "test1",
"display_name": "Test Instance 1",
"created_at": "2020-07-13T17:37:59.341728283Z",
"status": "active",
"cluster": "metrics-enterprise-dev",
"limits": {
"ingestion_rate": 350000,
"ingestion_rate_strategy": "local",
"ingestion_burst_size": 350000,
"accept_ha_samples": false,
"ha_cluster_label": "cluster",
"ha_replica_label": "__replica__",
"ha_max_clusters": 0,
"drop_labels": null,
"max_label_name_length": 1024,
"max_label_value_length": 2048,
"max_label_names_per_series": 30,
"max_metadata_length": 1024,
"reject_old_samples": false,
"reject_old_samples_max_age": 1209600000000000,
"creation_grace_period": 600000000000,
"enforce_metadata_metric_name": true,
"enforce_metric_name": true,
"ingestion_tenant_shard_size": 0,
"max_series_per_query": 42000,
"max_samples_per_query": 100000,
"max_series_per_user": 42000,
"max_series_per_metric": 0,
"max_global_series_per_user": 300000,
"max_global_series_per_metric": 300000,
"min_chunk_length": 0,
"max_metadata_per_user": 8000,
"max_metadata_per_metric": 10,
"max_global_metadata_per_user": 0,
"max_global_metadata_per_metric": 0,
"max_chunks_per_query": 2000000,
"max_query_lookback": "0s",
"max_query_length": 0,
"max_query_parallelism": 14,
"cardinality_limit": 100000,
"max_cache_freshness": 60000000000,
"max_queriers_per_tenant": 0,
"ruler_evaluation_delay_duration": 0,
"ruler_tenant_shard_size": 0,
"ruler_max_rules_per_rule_group": 0,
"ruler_max_rule_groups_per_tenant": 0,
"store_gateway_tenant_shard_size": 0,
"compactor_blocks_retention_period": 0,
"s3_sse_type": "",
"s3_sse_kms_key_id": "",
"s3_sse_kms_encryption_context": "",
"per_tenant_override_config": "",
"per_tenant_override_period": 10000000000
}
}Note that all available limit settings are populated, even ones that we did not include in our POST request. Any limit settings not included when a new instance is created with some limit settings are populated from the global default limits. This also applies when limits are added to an instance that previously did not have them: any limit settings that are not included will be populated from the global default limits.
Removing limits via the admin API
Limits can be removed from an instance using the admin API by modifying the instance.
Create a JSON payload, test1-update.json, that doesn’t include the limits field.
{
"display_name": "Test Instance 1",
"status": "active",
"cluster": "metrics-enterprise-dev"
}Use curl to PUT the data to the admin API.
$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/test1 \
-X PUT --data @test1-update.json | jq
{
"name": "test1",
"display_name": "Test Instance 1",
"created_at": "2020-07-13T17:37:59.341728283Z",
"status": "active",
"cluster": "metrics-enterprise-dev"
}The modified instance object is included in the update response.
Limit sources
Limits applied to an instance are only taken from a single location at a time:
- Per-instance limits set by the admin API.
- Per-instance limits set by a runtime configuration file.
- Global default limits set by a configuration file.
These locations are checked for limits to apply to operations, in that order. Once a location for limits to apply to an operation is found, no other locations are checked. The effect of this is that if you set limits for an instance using the admin API, any limit settings for that instance in a runtime configuration file will be ignored.
Implementation
Enforcing limits based on values set with the admin API requires periodically making requests to object storage (S3, GCS, etc.) to get the most up to date limits for each instance. This means that based on the period you have configured to refresh instance-based limits (1 minute by default), N GET requests and 1 LIST request will be made to object storage, where N is the number of instances in your Grafana Enterprise Metrics cluster.
Setting per tenant retention of metrics
To limit the amount of time a tenant can store metrics set compactor_blocks_retention_period for that tenant. The setting is a duration. It accepts values like 2y meaning 2 years and 60d meaning 60 days. After being marked for deletion the blocks are deleted by the blocks cleaner after a time indicated by the deletion_delay configuration setting (default=12h) has passed.
Example
Create a JSON payload, retention.json, for creating a new instance that includes a retention period.
{
"name": "retention",
"display_name": "With retention",
"status": "active",
"cluster": "metrics-enterprise-dev",
"limits": {
"compactor_blocks_retention_period": "1y"
}
}Use curl to POST the data to the admin API.
$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances \
-X POST --data @retention.jsonWas this page helpful?
Related resources from Grafana Labs
