Configure the Pyroscope server on Grafana Labs

About Grafana Pyroscope anonymous usage statistics reporting

Wed, 08 Apr 2026 14:38:28 +0000

About Grafana Pyroscope anonymous usage statistics reporting

By default, Pyroscope reports anonymous, non-sensitive, non-personally identifiable information about the running cluster to a remote statistics server. Pyroscope maintainers use this anonymous information to learn more about how the open source community runs Pyroscope and what the Pyroscope team should focus on when working on the next features and documentation improvements.

The anonymous usage statistics reporting is enabled by default. You can opt-out setting the CLI flag -usage-stats.enabled=false or its respective YAML configuration option.

The statistics server

When usage statistics reporting is enabled, information is collected by a server that Grafana Labs runs. Statistics are collected at https://stats.grafana.org.

Which information is collected

When the usage statistics reporting is enabled, Grafana Pyroscope collects the following information:

Information about the Pyroscope cluster and version:
- A unique, randomly-generated Pyroscope cluster identifier, such as 3749b5e2-b727-4107-95ae-172abac27496.
- The timestamp when the anonymous usage statistics reporting was enabled for the first time, and the cluster identifier was created.
- The Pyroscope version, such as 1.13.1.
- The Pyroscope branch, revision, and Golang version that was used to build the binary.
Information about the environment where Pyroscope is running:
- The operating system, such as linux.
- The architecture, such as amd64.
- The Pyroscope memory utilization and number of goroutines.
- The number of logical CPU cores available to the Pyroscope process.
Information about the Pyroscope configuration:
- The -target parameter value, such as all when running Pyroscope in monolithic mode.
- The -storage.backend value, such as s3.
- The -distributor.replication-factor value, such as 3.
Information about the Pyroscope cluster scale:
- Distributor:
  - Bytes received.
  - Profiles received with breakdown by profile type and programming language.
  - Profile sizes with breakdown by programming language.
- Ingester:
  - Number of active tenants.

Note
Pyroscope maintainers commit to keeping the list of tracked information updated over time, and reporting any change both via the CHANGELOG and the release notes.

Disable the anonymous usage statistics reporting

If possible, we ask you to keep the usage reporting feature enabled and help us understand more about how the open source community runs Pyroscope. In case you want to opt-out from anonymous usage statistics reporting, set the CLI flag -usage-stats.enabled=false or change the following YAML configuration:

analytics:
  reporting_enabled: false

About Pyroscope configurations

Wed, 08 Apr 2026 14:38:28 +0000

About Pyroscope configurations

You can configure Grafana Pyroscope using a (YAML-based) configuration file or CLI (command-line-interface) flags. You should specify your configuration using the configuration file rather than CLI flags. Every parameter that is set in the configuration file can also be set via a corresponding CLI flag. If you specify both CLI flags and configuration parameters, CLI flags take precedence over corresponding values in a configuration file. You can specify the configuration file by using the -config.file CLI flag.

To see the CLI flags that you need to get started with Pyroscope, run the pyroscope -help command.

To see the current configuration state of any component, use the /api/v1/status/config HTTP API endpoint.

Operational considerations

You should use a single configuration file and either pass it to all replicas of Pyroscope (if you are running multiple single-process Pyroscope replicas) or to all components of Pyroscope (if you are running Pyroscope as microservices). If you are running Pyroscope on Kubernetes, you can achieve this by storing the configuration file in a ConfigMap and mounting it in each Pyroscope container.

This helps avoid a common misconfiguration pitfall: while certain configuration parameters might look like they’re only needed by one type of component, they might in fact be used by multiple components. For example, the -distributor.replication-factor CLI flag is not only required by ingesters, but also by distributors and queriers.

By using a single configuration file, you ensure that each component gets all the configuration that it needs without needing to track which parameter belongs to which component. There is no harm in passing a configuration that is specific to one component (such as an ingester) to another component (such as a querier). In such a case, the configuration is simply ignored.

If necessary, you can use advanced CLI flags to override specific values on a particular Pyroscope component or replica. This is helpful if you want to change a parameter that’s specific to a certain component, without having to do a full restart of all other components.

The most common use case for CLI flags is to use the -target flag to run Pyroscope as microservices. By setting the -target CLI flag, all Pyroscope components share the same configuration file, but you can make them behave as a given component by specifying a -target command-line value, such as -target=ingester or -target=querier.

Tenant IDs

Wed, 08 Apr 2026 14:38:28 +0000

Tenant IDs

Grafana Pyroscope is a multi-tenant system where tenants can query profiles that include their tenant ID. Within a Grafana Pyroscope cluster, the tenant ID is the unique identifier of a tenant. The query takes the tenant ID from the X-Scope-OrgID parameter that exists in the HTTP header of each request, for example X-Scope-OrgID: <TENANT-ID>.

To push profiles to Pyroscope for a specific tenant, refer to Configure the Client.

By default, multi-tenancy is disabled, the tenant ID is ignored and all profiles are stored and retrieved with the same tenant (anonymous).

To enable multi-tenancy, add the multitenancy_enabled parameter to the Grafana Pyroscope configuration file and set it to true. Alternatively you can also use command line arguments to enable multi-tenancy, for example --auth.multitenancy-enabled=true.

Restrictions

Tenant IDs can’t be longer than 150 bytes or characters in length and can only include the following supported characters:

Alphanumeric characters
- 0-9
- a-z
- A-Z
Special characters
- Exclamation point (!)
- Hyphen (-)
- Underscore (_)
- Single period (.)
- Asterisk (*)
- Single quote (')
- Open parenthesis (()
- Close parenthesis ())

Note
For security reasons, . and .. aren’t valid tenant IDs. All other characters, including slashes and whitespace, aren’t supported.

Pyroscope configuration parameters

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope configuration parameters

You can configure Pyroscope by using a YAML file or via command-line flags that represent configuration parameters. To specify the YAML file, use the -config.file command-line option. If you specify both the command-line flags and YAML configuration parameters, the command-line flags take precedence over values in a YAML file.

To see the current configuration of any component, go to the /config HTTP API endpoint. Passwords are filtered out of this endpoint.

Parameters are written in YAML format, and brackets indicate that a parameter is optional.

Generic placeholders

<boolean>: a boolean that can take the values true or false
<int>: any integer matching the regular expression [1-9]+[0-9]*
<duration>: a duration matching the regular expression [0-9]+(ns|us|µs|ms|s|m|h|d|w|y) where y = 365 days
<string>: a string
<url>: a URL
<filepath>: a string containing an absolute or relative path and filename to a file on disk
<prefix>: a CLI flag prefix based on the context (look at the parent configuration block to see which CLI flags prefix should be used)
<relabel_config>: a Prometheus relabeling configuration
<time>: a timestamp, with available formats:
- 2006-01-20 (midnight, local timezone)
- 2006-01-20T15:04 (local timezone)
- RFC 3339 formats: 2006-01-20T15:04:05Z (UTC) or 2006-01-20T15:04:05+07:00 (explicit timezone)

Use environment variables in the configuration

You can use environment variable references in the YAML configuration file to set values that need to be configurable during deployment. To do this, pass -config.expand-env=true on the command line and use ${VAR}, where VAR is the name of the environment variable.

Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the YAML file is parsed. References to undefined variables are replaced by empty strings unless you specify a default value or custom error text.

To specify a default value, use ${VAR:-default_value}, where default_value is the value to use if the environment variable is undefined.

Configuration parameters

# Comma-separated list of Pyroscope modules to load. The alias 'all' can be used
# in the list to load a number of core modules and will enable single-binary
# mode.
# CLI flag: -target
[target: <string> | default = "all"]

api:
  # base URL for when the server is behind a reverse proxy with a different path
  # CLI flag: -api.base-url
  [base-url: <string> | default = ""]

# The server block configures the HTTP and gRPC server of the launched
# service(s).
[server: <server>]

# The distributor block configures the distributor.
[distributor: <distributor>]

# The querier block configures the querier.
[querier: <querier>]

# The query_frontend block configures the query-frontend.
[frontend: <query_frontend>]

# The frontend_worker block configures the frontend-worker.
[frontend_worker: <frontend_worker>]

# The limits block configures default and per-tenant limits imposed by
# components.
[limits: <limits>]

# The query_scheduler block configures the query-scheduler.
[query_scheduler: <query_scheduler>]

# The ingester block configures the ingester.
[ingester: <ingester>]

# The store_gateway block configures the store-gateway.
[store_gateway: <store_gateway>]

# The memberlist block configures the Gossip memberlist.
[memberlist: <memberlist>]

pyroscopedb:
  # Directory used for local storage.
  # CLI flag: -pyroscopedb.data-path
  [data_path: <string> | default = "./data"]

  # Upper limit to the duration of a Pyroscope block.
  # CLI flag: -pyroscopedb.max-block-duration
  [max_block_duration: <duration> | default = 1h]

  # How big should a single row group be uncompressed
  # CLI flag: -pyroscopedb.row-group-target-size
  [row_group_target_size: <int> | default = 1342177280]

  # Specifies the dimension by which symbols are partitioned. By default, the
  # partitioning is determined automatically.
  # CLI flag: -pyroscopedb.symbols-partition-label
  [symbols_partition_label: <string> | default = ""]

  # How much available disk space to keep in GiB
  # CLI flag: -pyroscopedb.retention-policy-min-free-disk-gb
  [min_free_disk_gb: <int> | default = 10]

  # Which percentage of free disk space to keep
  # CLI flag: -pyroscopedb.retention-policy-min-disk-available-percentage
  [min_disk_available_percentage: <float> | default = 0.05]

  # How often to enforce disk retention
  # CLI flag: -pyroscopedb.retention-policy-enforcement-interval
  [enforcement_interval: <duration> | default = 5m]

  # Disable retention policy enforcement
  # CLI flag: -pyroscopedb.retention-policy-disable
  [disable_enforcement: <boolean> | default = false]

tracing:
  # Set to false to disable tracing.
  # CLI flag: -tracing.enabled
  [enabled: <boolean> | default = true]

runtime_config:
  # How often to check runtime config files.
  # CLI flag: -runtime-config.reload-period
  [period: <duration> | default = 10s]

  # Comma separated list of yaml files with the configuration that can be
  # updated at runtime. Runtime config files will be merged from left to right.
  # CLI flag: -runtime-config.file
  [file: <string> | default = ""]

# The compactor block configures the compactor.
[compactor: <compactor>]

tenant_settings:
  recording_rules:
    # Enable the storing of recording rules in tenant settings.
    # CLI flag: -tenant-settings.recording-rules.enabled
    [enabled: <boolean> | default = false]

storage:
  # Backend storage to use. Supported backends are: s3, gcs, azure, swift,
  # filesystem, cos.
  # CLI flag: -storage.backend
  [backend: <string> | default = ""]

  # The s3_backend block configures the connection to Amazon S3 object storage
  # backend.
  [s3: <s3_storage_backend>]

  # The gcs_backend block configures the connection to Google Cloud Storage
  # object storage backend.
  [gcs: <gcs_storage_backend>]

  # The azure_storage_backend block configures the connection to Azure object
  # storage backend.
  [azure: <azure_storage_backend>]

  # The swift_storage_backend block configures the connection to OpenStack
  # Object Storage (Swift) object storage backend.
  [swift: <swift_storage_backend>]

  cos:
    # COS bucket name
    # CLI flag: -storage.cos.bucket
    [bucket: <string> | default = ""]

    # COS region name
    # CLI flag: -storage.cos.region
    [region: <string> | default = ""]

    # COS app id
    # CLI flag: -storage.cos.app-id
    [app_id: <string> | default = ""]

    # COS storage endpoint
    # CLI flag: -storage.cos.endpoint
    [endpoint: <string> | default = ""]

    # COS secret key
    # CLI flag: -storage.cos.secret-key
    [secret_key: <string> | default = ""]

    # COS secret id
    # CLI flag: -storage.cos.secret-id
    [secret_id: <string> | default = ""]

    http:
      # The time an idle connection will remain idle before closing.
      # CLI flag: -storage.cos.http.idle-conn-timeout
      [idle_conn_timeout: <duration> | default = 1m30s]

      # The amount of time the client will wait for a servers response headers.
      # CLI flag: -storage.cos.http.response-header-timeout
      [response_header_timeout: <duration> | default = 2m]

      # If the client connects to COS via HTTPS and this option is enabled, the
      # client will accept any certificate and hostname.
      # CLI flag: -storage.cos.http.insecure-skip-verify
      [insecure_skip_verify: <boolean> | default = false]

      # Maximum time to wait for a TLS handshake. 0 means no limit.
      # CLI flag: -storage.cos.tls-handshake-timeout
      [tls_handshake_timeout: <duration> | default = 10s]

      # The time to wait for a server's first response headers after fully
      # writing the request headers if the request has an Expect header. 0 to
      # send the request body immediately.
      # CLI flag: -storage.cos.expect-continue-timeout
      [expect_continue_timeout: <duration> | default = 1s]

      # Maximum number of idle (keep-alive) connections across all hosts. 0
      # means no limit.
      # CLI flag: -storage.cos.max-idle-connections
      [max_idle_connections: <int> | default = 100]

      # Maximum number of idle (keep-alive) connections to keep per-host. If 0,
      # a built-in default value is used.
      # CLI flag: -storage.cos.max-idle-connections-per-host
      [max_idle_connections_per_host: <int> | default = 100]

      # Maximum number of connections per host. 0 means no limit.
      # CLI flag: -storage.cos.max-connections-per-host
      [max_connections_per_host: <int> | default = 0]

  # The filesystem_storage_backend block configures the usage of local file
  # system as object storage backend.
  [filesystem: <filesystem_storage_backend>]

  # Prefix for all objects stored in the backend storage. For simplicity, it may
  # only contain digits and English alphabet characters, hyphens, underscores,
  # dots and forward slashes.
  # CLI flag: -storage.prefix
  [prefix: <string> | default = ""]

  # Deprecated: Use 'storage..prefix' instead. Prefix for all objects stored in
  # the backend storage. For simplicity, it may only contain digits and English
  # alphabet characters, hyphens, underscores, dots and forward slashes.
  # CLI flag: -storage.storage-prefix
  [storage_prefix: <string> | default = ""]

self_profiling:
  # When running in single binary (--target=all) Pyroscope will push (Go SDK)
  # profiles to itself. Set to true to disable self-profiling.
  # CLI flag: -self-profiling.disable-push
  [disable_push: <boolean> | default = false]

  # CLI flag: -self-profiling.mutex-profile-fraction
  [mutex_profile_fraction: <int> | default = 5]

  # CLI flag: -self-profiling.block-profile-rate
  [block_profile_rate: <int> | default = 5]

  # Read k6 labels from request headers and set them as dynamic profile tags.
  # CLI flag: -self-profiling.use-k6-middleware
  [use_k6_middleware: <boolean> | default = false]

# When set to true, incoming HTTP requests must specify tenant ID in HTTP
# X-Scope-OrgId header. When set to false, tenant ID anonymous is used instead.
# CLI flag: -auth.multitenancy-enabled
[multitenancy_enabled: <boolean> | default = false]

# The analytics block configures usage statistics collection. For more details
# about usage statistics, refer to [Anonymous usage statistics
# reporting](../anonymous-usage-statistics-reporting)
[analytics: <analytics>]

# Prints the application banner at startup.
# CLI flag: -config.show_banner
[show_banner: <boolean> | default = true]

# Wait time before shutting down after a termination signal.
# CLI flag: -shutdown-delay
[shutdown_delay: <duration> | default = 0s]

embedded_grafana:
  # The directory where the Grafana data will be stored.
  # CLI flag: -embedded-grafana.data-path
  [data_path: <string> | default = "./data/__embedded_grafana/"]

  # The port on which the Grafana will listen.
  # CLI flag: -embedded-grafana.listen-port
  [listen_port: <int> | default = 4041]

  # The URL of the Pyroscope instance to use for the Grafana datasources.
  # CLI flag: -embedded-grafana.pyroscope-url
  [pyroscope_url: <string> | default = "http://localhost:4040"]

server

The server block configures the HTTP and gRPC server of the launched service(s).

# HTTP server listen network, default tcp
# CLI flag: -server.http-listen-network
[http_listen_network: <string> | default = "tcp"]

# HTTP server listen address.
# CLI flag: -server.http-listen-address
[http_listen_address: <string> | default = ""]

# HTTP server listen port.
# CLI flag: -server.http-listen-port
[http_listen_port: <int> | default = 4040]

# Maximum number of simultaneous http connections, <=0 to disable
# CLI flag: -server.http-conn-limit
[http_listen_conn_limit: <int> | default = 0]

# gRPC server listen network
# CLI flag: -server.grpc-listen-network
[grpc_listen_network: <string> | default = "tcp"]

# gRPC server listen address.
# CLI flag: -server.grpc-listen-address
[grpc_listen_address: <string> | default = ""]

# gRPC server listen port.
# CLI flag: -server.grpc-listen-port
[grpc_listen_port: <int> | default = 9095]

# Maximum number of simultaneous grpc connections, <=0 to disable
# CLI flag: -server.grpc-conn-limit
[grpc_listen_conn_limit: <int> | default = 0]

# If true, the max streams by connection gauge will be collected.
# CLI flag: -server.grpc-collect-max-streams-by-conn
[grpc_collect_max_streams_by_conn: <boolean> | default = true]

# Enables PROXY protocol.
# CLI flag: -server.proxy-protocol-enabled
[proxy_protocol_enabled: <boolean> | default = false]

# Comma-separated list of cipher suites to use. If blank, the default Go cipher
# suites is used.
# CLI flag: -server.tls-cipher-suites
[tls_cipher_suites: <string> | default = ""]

# Minimum TLS version to use. Allowed values: VersionTLS10, VersionTLS11,
# VersionTLS12, VersionTLS13. If blank, the Go TLS minimum version is used.
# CLI flag: -server.tls-min-version
[tls_min_version: <string> | default = ""]

http_tls_config:
  # Server TLS certificate. This configuration parameter is YAML only.
  [cert: <string> | default = ""]

  # Server TLS key. This configuration parameter is YAML only.
  [key: <string> | default = ""]

  # Root certificate authority used to verify client certificates. This
  # configuration parameter is YAML only.
  [client_ca: <string> | default = ""]

  # HTTP server cert path.
  # CLI flag: -server.http-tls-cert-path
  [cert_file: <string> | default = ""]

  # HTTP server key path.
  # CLI flag: -server.http-tls-key-path
  [key_file: <string> | default = ""]

  # HTTP TLS Client Auth type.
  # CLI flag: -server.http-tls-client-auth
  [client_auth_type: <string> | default = ""]

  # HTTP TLS Client CA path.
  # CLI flag: -server.http-tls-ca-path
  [client_ca_file: <string> | default = ""]

grpc_tls_config:
  # Server TLS certificate. This configuration parameter is YAML only.
  [cert: <string> | default = ""]

  # Server TLS key. This configuration parameter is YAML only.
  [key: <string> | default = ""]

  # Root certificate authority used to verify client certificates. This
  # configuration parameter is YAML only.
  [client_ca: <string> | default = ""]

  # GRPC TLS server cert path.
  # CLI flag: -server.grpc-tls-cert-path
  [cert_file: <string> | default = ""]

  # GRPC TLS server key path.
  # CLI flag: -server.grpc-tls-key-path
  [key_file: <string> | default = ""]

  # GRPC TLS Client Auth type.
  # CLI flag: -server.grpc-tls-client-auth
  [client_auth_type: <string> | default = ""]

  # GRPC TLS Client CA path.
  # CLI flag: -server.grpc-tls-ca-path
  [client_ca_file: <string> | default = ""]

# Register the intrumentation handlers (/metrics etc).
# CLI flag: -server.register-instrumentation
[register_instrumentation: <boolean> | default = true]

# If set to true, gRPC statuses will be reported in instrumentation labels with
# their string representations. Otherwise, they will be reported as "error".
# CLI flag: -server.report-grpc-codes-in-instrumentation-label-enabled
[report_grpc_codes_in_instrumentation_label_enabled: <boolean> | default = false]

# Timeout for graceful shutdowns
# CLI flag: -server.graceful-shutdown-timeout
[graceful_shutdown_timeout: <duration> | default = 30s]

# Read timeout for entire HTTP request, including headers and body.
# CLI flag: -server.http-read-timeout
[http_server_read_timeout: <duration> | default = 30s]

# Read timeout for HTTP request headers. If set to 0, value of
# -server.http-read-timeout is used.
# CLI flag: -server.http-read-header-timeout
[http_server_read_header_timeout: <duration> | default = 0s]

# Write timeout for HTTP server
# CLI flag: -server.http-write-timeout
[http_server_write_timeout: <duration> | default = 30s]

# Idle timeout for HTTP server
# CLI flag: -server.http-idle-timeout
[http_server_idle_timeout: <duration> | default = 2m]

# Log closed connections that did not receive any response, most likely because
# client didn't send any request within timeout.
# CLI flag: -server.http-log-closed-connections-without-response-enabled
[http_log_closed_connections_without_response_enabled: <boolean> | default = false]

# Limit on the size of a gRPC message this server can receive (bytes).
# CLI flag: -server.grpc-max-recv-msg-size-bytes
[grpc_server_max_recv_msg_size: <int> | default = 4194304]

# Limit on the size of a gRPC message this server can send (bytes).
# CLI flag: -server.grpc-max-send-msg-size-bytes
[grpc_server_max_send_msg_size: <int> | default = 4194304]

# Limit on the number of concurrent streams for gRPC calls per client connection
# (0 = unlimited)
# CLI flag: -server.grpc-max-concurrent-streams
[grpc_server_max_concurrent_streams: <int> | default = 100]

# The duration after which an idle connection should be closed. Default:
# infinity
# CLI flag: -server.grpc.keepalive.max-connection-idle
[grpc_server_max_connection_idle: <duration> | default = 2562047h47m16.854775807s]

# The duration for the maximum amount of time a connection may exist before it
# will be closed. Default: infinity
# CLI flag: -server.grpc.keepalive.max-connection-age
[grpc_server_max_connection_age: <duration> | default = 2562047h47m16.854775807s]

# An additive period after max-connection-age after which the connection will be
# forcibly closed. Default: infinity
# CLI flag: -server.grpc.keepalive.max-connection-age-grace
[grpc_server_max_connection_age_grace: <duration> | default = 2562047h47m16.854775807s]

# Duration after which a keepalive probe is sent in case of no activity over the
# connection., Default: 2h
# CLI flag: -server.grpc.keepalive.time
[grpc_server_keepalive_time: <duration> | default = 2h]

# After having pinged for keepalive check, the duration after which an idle
# connection should be closed, Default: 20s
# CLI flag: -server.grpc.keepalive.timeout
[grpc_server_keepalive_timeout: <duration> | default = 20s]

# Minimum amount of time a client should wait before sending a keepalive ping.
# If client sends keepalive ping more often, server will send GOAWAY and close
# the connection.
# CLI flag: -server.grpc.keepalive.min-time-between-pings
[grpc_server_min_time_between_pings: <duration> | default = 5m]

# If true, server allows keepalive pings even when there are no active
# streams(RPCs). If false, and client sends ping when there are no active
# streams, server will send GOAWAY and close the connection.
# CLI flag: -server.grpc.keepalive.ping-without-stream-allowed
[grpc_server_ping_without_stream_allowed: <boolean> | default = false]

# If non-zero, configures the amount of GRPC server workers used to serve the
# requests.
# CLI flag: -server.grpc.num-workers
[grpc_server_num_workers: <int> | default = 0]

# If true, the request_message_bytes, response_message_bytes, and
# inflight_requests metrics will be tracked. Enabling this option prevents the
# use of memory pools for parsing gRPC request bodies and may lead to more
# memory allocations.
# CLI flag: -server.grpc.stats-tracking-enabled
[grpc_server_stats_tracking_enabled: <boolean> | default = true]

# Deprecated option, has no effect and will be removed in a future version.
# CLI flag: -server.grpc.recv-buffer-pools-enabled
[grpc_server_recv_buffer_pools_enabled: <boolean> | default = false]

# Output log messages in the given format. Valid formats: [logfmt, json]
# CLI flag: -log.format
[log_format: <string> | default = "logfmt"]

# Only log messages with the given severity or above. Valid levels: [debug,
# info, warn, error]
# CLI flag: -log.level
[log_level: <string> | default = "info"]

# Optionally log the source IPs.
# CLI flag: -server.log-source-ips-enabled
[log_source_ips_enabled: <boolean> | default = false]

# Log all source IPs instead of only the originating one. Only used if
# server.log-source-ips-enabled is true
# CLI flag: -server.log-source-ips-full
[log_source_ips_full: <boolean> | default = false]

# Header field storing the source IPs. Only used if
# server.log-source-ips-enabled is true. If not set the default Forwarded,
# X-Real-IP and X-Forwarded-For headers are used
# CLI flag: -server.log-source-ips-header
[log_source_ips_header: <string> | default = ""]

# Regex for matching the source IPs. Only used if server.log-source-ips-enabled
# is true. If not set the default Forwarded, X-Real-IP and X-Forwarded-For
# headers are used
# CLI flag: -server.log-source-ips-regex
[log_source_ips_regex: <string> | default = ""]

# Optionally log request headers.
# CLI flag: -server.log-request-headers
[log_request_headers: <boolean> | default = false]

# Optionally log requests at info level instead of debug level. Applies to
# request headers as well if server.log-request-headers is enabled.
# CLI flag: -server.log-request-at-info-level-enabled
[log_request_at_info_level_enabled: <boolean> | default = false]

# Comma separated list of headers to exclude from loggin. Only used if
# server.log-request-headers is true.
# CLI flag: -server.log-request-headers-exclude-list
[log_request_exclude_headers_list: <string> | default = ""]

# Optionally add request headers to tracing spans.
# CLI flag: -server.trace-request-headers
[trace_request_headers: <boolean> | default = false]

# Comma separated list of headers to exclude from tracing spans. Only used if
# server.trace-request-headers is true. The following headers are always
# excluded: Authorization, Cookie, X-Csrf-Token.
# CLI flag: -server.trace-request-headers-exclude-list
[trace_request_exclude_headers_list: <string> | default = ""]

# Base path to serve all API routes from (e.g. /v1/)
# CLI flag: -server.path-prefix
[http_path_prefix: <string> | default = ""]

cluster_validation:
  # Optionally define the cluster validation label.
  # CLI flag: -server.cluster-validation.label
  [label: <string> | default = ""]

  grpc:
    # When enabled, cluster label validation is executed: configured cluster
    # validation label is compared with the cluster validation label received
    # through the requests.
    # CLI flag: -server.cluster-validation.grpc.enabled
    [enabled: <boolean> | default = false]

    # When enabled, soft cluster label validation is executed. Can be enabled
    # only together with server.cluster-validation.grpc.enabled
    # CLI flag: -server.cluster-validation.grpc.soft-validation
    [soft_validation: <boolean> | default = false]

  http:
    # When enabled, cluster label validation is executed: configured cluster
    # validation label is compared with the cluster validation label received
    # through the requests.
    # CLI flag: -server.cluster-validation.http.enabled
    [enabled: <boolean> | default = false]

    # When enabled, soft cluster label validation is executed. Can be enabled
    # only together with server.cluster-validation.http.enabled
    # CLI flag: -server.cluster-validation.http.soft-validation
    [soft_validation: <boolean> | default = false]

    # Comma-separated list of url paths that are excluded from the cluster
    # validation check.
    # CLI flag: -server.cluster-validation.http.excluded-paths
    [excluded_paths: <string> | default = ""]

    # Comma-separated list of user agents that are excluded from the cluster
    # validation check.
    # CLI flag: -server.cluster-validation.http.excluded-user-agents
    [excluded_user_agents: <string> | default = ""]

distributor

The distributor block configures the distributor.

# Timeout when pushing data to ingester.
# CLI flag: -distributor.push.timeout
[pushtimeout: <duration> | default = 5s]

pool_config:
  # How frequently to clean up clients for ingesters that have gone away.
  # CLI flag: -distributor.client-cleanup-period
  [client_cleanup_period: <duration> | default = 15s]

  # Run a health check on each ingester client during periodic cleanup.
  # CLI flag: -distributor.health-check-ingesters
  [health_check_ingesters: <boolean> | default = true]

  # Timeout for ingester client healthcheck RPCs.
  # CLI flag: -distributor.health-check-timeout
  [remote_timeout: <duration> | default = 5s]

ring:
  # The key-value store used to share the hash ring across multiple instances.
  kvstore:
    # Backend storage to use for the ring. Supported values are: consul, etcd,
    # inmemory, memberlist, multi.
    # CLI flag: -distributor.ring.store
    [store: <string> | default = "memberlist"]

    # The prefix for the keys in the store. Should end with a /.
    # CLI flag: -distributor.ring.prefix
    [prefix: <string> | default = "collectors/"]

    consul:
      # Hostname and port of Consul.
      # CLI flag: -distributor.ring.consul.hostname
      [host: <string> | default = "localhost:8500"]

      # ACL Token used to interact with Consul.
      # CLI flag: -distributor.ring.consul.acl-token
      [acl_token: <string> | default = ""]

      # HTTP timeout when talking to Consul
      # CLI flag: -distributor.ring.consul.client-timeout
      [http_client_timeout: <duration> | default = 20s]

      # Enable consistent reads to Consul.
      # CLI flag: -distributor.ring.consul.consistent-reads
      [consistent_reads: <boolean> | default = false]

      # Rate limit when watching key or prefix in Consul, in requests per
      # second. 0 disables the rate limit.
      # CLI flag: -distributor.ring.consul.watch-rate-limit
      [watch_rate_limit: <float> | default = 1]

      # Burst size used in rate limit. Values less than 1 are treated as 1.
      # CLI flag: -distributor.ring.consul.watch-burst-size
      [watch_burst_size: <int> | default = 1]

      # Maximum duration to wait before retrying a Compare And Swap (CAS)
      # operation.
      # CLI flag: -distributor.ring.consul.cas-retry-delay
      [cas_retry_delay: <duration> | default = 1s]

    etcd:
      # The etcd endpoints to connect to.
      # CLI flag: -distributor.ring.etcd.endpoints
      [endpoints: <list of strings> | default = []]

      # The dial timeout for the etcd connection.
      # CLI flag: -distributor.ring.etcd.dial-timeout
      [dial_timeout: <duration> | default = 10s]

      # The maximum number of retries to do for failed ops.
      # CLI flag: -distributor.ring.etcd.max-retries
      [max_retries: <int> | default = 10]

      # Enable TLS.
      # CLI flag: -distributor.ring.etcd.tls-enabled
      [tls_enabled: <boolean> | default = false]

      # Path to the client certificate, which will be used for authenticating
      # with the server. Also requires the key path to be configured.
      # CLI flag: -distributor.ring.etcd.tls-cert-path
      [tls_cert_path: <string> | default = ""]

      # Path to the key for the client certificate. Also requires the client
      # certificate to be configured.
      # CLI flag: -distributor.ring.etcd.tls-key-path
      [tls_key_path: <string> | default = ""]

      # Path to the CA certificates to validate server certificate against. If
      # not set, the host's root CA certificates are used.
      # CLI flag: -distributor.ring.etcd.tls-ca-path
      [tls_ca_path: <string> | default = ""]

      # Override the expected name on the server certificate.
      # CLI flag: -distributor.ring.etcd.tls-server-name
      [tls_server_name: <string> | default = ""]

      # Skip validating server certificate.
      # CLI flag: -distributor.ring.etcd.tls-insecure-skip-verify
      [tls_insecure_skip_verify: <boolean> | default = false]

      # Override the default cipher suite list (separated by commas). Allowed
      # values:
      # 
      # Secure Ciphers:
      # - TLS_AES_128_GCM_SHA256
      # - TLS_AES_256_GCM_SHA384
      # - TLS_CHACHA20_POLY1305_SHA256
      # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
      # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
      # - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      # - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      # - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
      # 
      # Insecure Ciphers:
      # - TLS_RSA_WITH_RC4_128_SHA
      # - TLS_RSA_WITH_3DES_EDE_CBC_SHA
      # - TLS_RSA_WITH_AES_128_CBC_SHA
      # - TLS_RSA_WITH_AES_256_CBC_SHA
      # - TLS_RSA_WITH_AES_128_CBC_SHA256
      # - TLS_RSA_WITH_AES_128_GCM_SHA256
      # - TLS_RSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
      # - TLS_ECDHE_RSA_WITH_RC4_128_SHA
      # - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
      # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
      # CLI flag: -distributor.ring.etcd.tls-cipher-suites
      [tls_cipher_suites: <string> | default = ""]

      # Override the default minimum TLS version. Allowed values: VersionTLS10,
      # VersionTLS11, VersionTLS12, VersionTLS13
      # CLI flag: -distributor.ring.etcd.tls-min-version
      [tls_min_version: <string> | default = ""]

      # Etcd username.
      # CLI flag: -distributor.ring.etcd.username
      [username: <string> | default = ""]

      # Etcd password.
      # CLI flag: -distributor.ring.etcd.password
      [password: <string> | default = ""]

    multi:
      # Primary backend storage used by multi-client.
      # CLI flag: -distributor.ring.multi.primary
      [primary: <string> | default = ""]

      # Secondary backend storage used by multi-client.
      # CLI flag: -distributor.ring.multi.secondary
      [secondary: <string> | default = ""]

      # Mirror writes to secondary store.
      # CLI flag: -distributor.ring.multi.mirror-enabled
      [mirror_enabled: <boolean> | default = false]

      # Timeout for storing value to secondary store.
      # CLI flag: -distributor.ring.multi.mirror-timeout
      [mirror_timeout: <duration> | default = 2s]

  # Period at which to heartbeat to the ring. 0 = disabled.
  # CLI flag: -distributor.ring.heartbeat-period
  [heartbeat_period: <duration> | default = 15s]

  # The heartbeat timeout after which distributors are considered unhealthy
  # within the ring. 0 = never (timeout disabled).
  # CLI flag: -distributor.ring.heartbeat-timeout
  [heartbeat_timeout: <duration> | default = 1m]

  # Instance ID to register in the ring.
  # CLI flag: -distributor.ring.instance-id
  [instance_id: <string> | default = "<hostname>"]

  # List of network interface names to look up when finding the instance IP
  # address.
  # CLI flag: -distributor.ring.instance-interface-names
  [instance_interface_names: <list of strings> | default = [<private network interfaces>]]

  # Port to advertise in the ring (defaults to -server.http-listen-port).
  # CLI flag: -distributor.ring.instance-port
  [instance_port: <int> | default = 0]

  # IP address to advertise in the ring. Default is auto-detected.
  # CLI flag: -distributor.ring.instance-addr
  [instance_addr: <string> | default = ""]

  # Enable using a IPv6 instance address. (default false)
  # CLI flag: -distributor.ring.instance-enable-ipv6
  [instance_enable_ipv6: <boolean> | default = false]

ingester

The ingester block configures the ingester.

lifecycler:
  ring:
    kvstore:
      # Backend storage to use for the ring. Supported values are: consul, etcd,
      # inmemory, memberlist, multi.
      # CLI flag: -ring.store
      [store: <string> | default = "consul"]

      # The prefix for the keys in the store. Should end with a /.
      # CLI flag: -ring.prefix
      [prefix: <string> | default = "collectors/"]

      consul:
        # Hostname and port of Consul.
        # CLI flag: -consul.hostname
        [host: <string> | default = "localhost:8500"]

        # ACL Token used to interact with Consul.
        # CLI flag: -consul.acl-token
        [acl_token: <string> | default = ""]

        # HTTP timeout when talking to Consul
        # CLI flag: -consul.client-timeout
        [http_client_timeout: <duration> | default = 20s]

        # Enable consistent reads to Consul.
        # CLI flag: -consul.consistent-reads
        [consistent_reads: <boolean> | default = false]

        # Rate limit when watching key or prefix in Consul, in requests per
        # second. 0 disables the rate limit.
        # CLI flag: -consul.watch-rate-limit
        [watch_rate_limit: <float> | default = 1]

        # Burst size used in rate limit. Values less than 1 are treated as 1.
        # CLI flag: -consul.watch-burst-size
        [watch_burst_size: <int> | default = 1]

        # Maximum duration to wait before retrying a Compare And Swap (CAS)
        # operation.
        # CLI flag: -consul.cas-retry-delay
        [cas_retry_delay: <duration> | default = 1s]

      etcd:
        # The etcd endpoints to connect to.
        # CLI flag: -etcd.endpoints
        [endpoints: <list of strings> | default = []]

        # The dial timeout for the etcd connection.
        # CLI flag: -etcd.dial-timeout
        [dial_timeout: <duration> | default = 10s]

        # The maximum number of retries to do for failed ops.
        # CLI flag: -etcd.max-retries
        [max_retries: <int> | default = 10]

        # Enable TLS.
        # CLI flag: -etcd.tls-enabled
        [tls_enabled: <boolean> | default = false]

        # Path to the client certificate, which will be used for authenticating
        # with the server. Also requires the key path to be configured.
        # CLI flag: -etcd.tls-cert-path
        [tls_cert_path: <string> | default = ""]

        # Path to the key for the client certificate. Also requires the client
        # certificate to be configured.
        # CLI flag: -etcd.tls-key-path
        [tls_key_path: <string> | default = ""]

        # Path to the CA certificates to validate server certificate against. If
        # not set, the host's root CA certificates are used.
        # CLI flag: -etcd.tls-ca-path
        [tls_ca_path: <string> | default = ""]

        # Override the expected name on the server certificate.
        # CLI flag: -etcd.tls-server-name
        [tls_server_name: <string> | default = ""]

        # Skip validating server certificate.
        # CLI flag: -etcd.tls-insecure-skip-verify
        [tls_insecure_skip_verify: <boolean> | default = false]

        # Override the default cipher suite list (separated by commas). Allowed
        # values:
        # 
        # Secure Ciphers:
        # - TLS_AES_128_GCM_SHA256
        # - TLS_AES_256_GCM_SHA384
        # - TLS_CHACHA20_POLY1305_SHA256
        # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        # - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        # - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        # - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        # - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        # - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        # - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        # - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        # 
        # Insecure Ciphers:
        # - TLS_RSA_WITH_RC4_128_SHA
        # - TLS_RSA_WITH_3DES_EDE_CBC_SHA
        # - TLS_RSA_WITH_AES_128_CBC_SHA
        # - TLS_RSA_WITH_AES_256_CBC_SHA
        # - TLS_RSA_WITH_AES_128_CBC_SHA256
        # - TLS_RSA_WITH_AES_128_GCM_SHA256
        # - TLS_RSA_WITH_AES_256_GCM_SHA384
        # - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
        # - TLS_ECDHE_RSA_WITH_RC4_128_SHA
        # - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
        # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        # CLI flag: -etcd.tls-cipher-suites
        [tls_cipher_suites: <string> | default = ""]

        # Override the default minimum TLS version. Allowed values:
        # VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
        # CLI flag: -etcd.tls-min-version
        [tls_min_version: <string> | default = ""]

        # Etcd username.
        # CLI flag: -etcd.username
        [username: <string> | default = ""]

        # Etcd password.
        # CLI flag: -etcd.password
        [password: <string> | default = ""]

      multi:
        # Primary backend storage used by multi-client.
        # CLI flag: -multi.primary
        [primary: <string> | default = ""]

        # Secondary backend storage used by multi-client.
        # CLI flag: -multi.secondary
        [secondary: <string> | default = ""]

        # Mirror writes to secondary store.
        # CLI flag: -multi.mirror-enabled
        [mirror_enabled: <boolean> | default = false]

        # Timeout for storing value to secondary store.
        # CLI flag: -multi.mirror-timeout
        [mirror_timeout: <duration> | default = 2s]

    # The heartbeat timeout after which ingesters are skipped for reads/writes.
    # 0 = never (timeout disabled).
    # CLI flag: -ring.heartbeat-timeout
    [heartbeat_timeout: <duration> | default = 1m]

    # The number of ingesters to write to and read from.
    # CLI flag: -distributor.replication-factor
    [replication_factor: <int> | default = 1]

    # True to enable the zone-awareness and replicate ingested samples across
    # different availability zones.
    # CLI flag: -distributor.zone-awareness-enabled
    [zone_awareness_enabled: <boolean> | default = false]

    # Comma-separated list of zones to exclude from the ring. Instances in
    # excluded zones will be filtered out from the ring.
    # CLI flag: -distributor.excluded-zones
    [excluded_zones: <string> | default = ""]

  # Number of tokens for each ingester.
  # CLI flag: -ingester.num-tokens
  [num_tokens: <int> | default = 128]

  # Period at which to heartbeat to consul. 0 = disabled.
  # CLI flag: -ingester.heartbeat-period
  [heartbeat_period: <duration> | default = 5s]

  # Heartbeat timeout after which instance is assumed to be unhealthy. 0 =
  # disabled.
  # CLI flag: -ingester.heartbeat-timeout
  [heartbeat_timeout: <duration> | default = 1m]

  # Observe tokens after generating to resolve collisions. Useful when using
  # gossiping ring.
  # CLI flag: -ingester.observe-period
  [observe_period: <duration> | default = 0s]

  # Period to wait for a claim from another member; will join automatically
  # after this.
  # CLI flag: -ingester.join-after
  [join_after: <duration> | default = 0s]

  # Minimum duration to wait after the internal readiness checks have passed but
  # before succeeding the readiness endpoint. This is used to slowdown
  # deployment controllers (eg. Kubernetes) after an instance is ready and
  # before they proceed with a rolling update, to give the rest of the cluster
  # instances enough time to receive ring updates.
  # CLI flag: -ingester.min-ready-duration
  [min_ready_duration: <duration> | default = 15s]

  # Name of network interface to read address from.
  # CLI flag: -ingester.lifecycler.interface
  [interface_names: <list of strings> | default = [<private network interfaces>]]

  # Enable IPv6 support. Required to make use of IP addresses from IPv6
  # interfaces.
  # CLI flag: -ingester.enable-inet6
  [enable_inet6: <boolean> | default = false]

  # Duration to sleep for before exiting, to ensure metrics are scraped.
  # CLI flag: -ingester.final-sleep
  [final_sleep: <duration> | default = 0s]

  # File path where tokens are stored. If empty, tokens are not stored at
  # shutdown and restored at startup.
  # CLI flag: -ingester.tokens-file-path
  [tokens_file_path: <string> | default = ""]

  # The availability zone where this instance is running.
  # CLI flag: -ingester.availability-zone
  [availability_zone: <string> | default = ""]

  # Unregister from the ring upon clean shutdown. It can be useful to disable
  # for rolling restarts with consistent naming in conjunction with
  # -distributor.extend-writes=false.
  # CLI flag: -ingester.unregister-on-shutdown
  [unregister_on_shutdown: <boolean> | default = true]

  # When enabled the readiness probe succeeds only after all instances are
  # ACTIVE and healthy in the ring, otherwise only the instance itself is
  # checked. This option should be disabled if in your cluster multiple
  # instances can be rolled out simultaneously, otherwise rolling updates may be
  # slowed down.
  # CLI flag: -ingester.readiness-check-ring-health
  [readiness_check_ring_health: <boolean> | default = true]

  # IP address to advertise in the ring.
  # CLI flag: -ingester.lifecycler.addr
  [address: <string> | default = ""]

  # port to advertise in consul (defaults to server.grpc-listen-port).
  # CLI flag: -ingester.lifecycler.port
  [port: <int> | default = 0]

  # ID to register in the ring.
  # CLI flag: -ingester.lifecycler.ID
  [id: <string> | default = "<hostname>"]

querier

The querier block configures the querier.

pool_config:
  # How frequently to clean up clients for ingesters that have gone away.
  # CLI flag: -querier.client-cleanup-period
  [client_cleanup_period: <duration> | default = 15s]

  # Run a health check on each ingester client during periodic cleanup.
  # CLI flag: -querier.health-check-ingesters
  [health_check_ingesters: <boolean> | default = true]

  # Timeout for ingester client healthcheck RPCs.
  # CLI flag: -querier.health-check-timeout
  [remote_timeout: <duration> | default = 5s]

# The time after which a metric should be queried from storage and not just
# ingesters. 0 means all queries are sent to store. If this option is enabled,
# the time range of the query sent to the store-gateway will be manipulated to
# ensure the query end is not more recent than 'now - query-store-after'.
# CLI flag: -querier.query-store-after
[query_store_after: <duration> | default = 4h]

query_frontend

The query_frontend block configures the query-frontend.

# Number of concurrent workers forwarding queries to single query-scheduler.
# CLI flag: -query-frontend.scheduler-worker-concurrency
[scheduler_worker_concurrency: <int> | default = 5]

# Configures the gRPC client used to communicate between the query-frontends and
# the query-schedulers.
# The CLI flags prefix for this block configuration is:
# query-frontend.grpc-client-config
[grpc_client_config: <grpc_client>]

# List of network interface names to look up when finding the instance IP
# address. This address is sent to query-scheduler and querier, which uses it to
# send the query response back to query-frontend.
# CLI flag: -query-frontend.instance-interface-names
[instance_interface_names: <list of strings> | default = [<private network interfaces>]]

# IP address to advertise to the querier (via scheduler) (default is
# auto-detected from network interfaces).
# CLI flag: -query-frontend.instance-addr
[instance_addr: <string> | default = ""]

# Enable using a IPv6 instance address. (default false)
# CLI flag: -query-frontend.instance-enable-ipv6
[instance_enable_ipv6: <boolean> | default = false]

# Port to advertise to query-scheduler and querier (defaults to
# -server.http-listen-port).
# CLI flag: -query-frontend.instance-port
[instance_port: <int> | default = 0]

frontend_worker

The frontend_worker block configures the frontend-worker.

# Querier ID, sent to the query-frontend to identify requests from the same
# querier. Defaults to hostname.
# CLI flag: -querier.id
[id: <string> | default = ""]

# Configures the gRPC client used to communicate between the queriers and the
# query-frontends / query-schedulers.
# The CLI flags prefix for this block configuration is: querier.frontend-client
[grpc_client_config: <grpc_client>]

# The maximum number of concurrent queries allowed.
# CLI flag: -querier.max-concurrent
[max_concurrent: <int> | default = 4]

query_scheduler

The query_scheduler block configures the query-scheduler.

# Maximum number of outstanding requests per tenant per query-scheduler.
# In-flight requests above this limit will fail with HTTP response status code
# 429.
# CLI flag: -query-scheduler.max-outstanding-requests-per-tenant
[max_outstanding_requests_per_tenant: <int> | default = 100]

# If a querier disconnects without sending notification about graceful shutdown,
# the query-scheduler will keep the querier in the tenant's shard until the
# forget delay has passed. This feature is useful to reduce the blast radius
# when shuffle-sharding is enabled.
# CLI flag: -query-scheduler.querier-forget-delay
[querier_forget_delay: <duration> | default = 0s]

# This configures the gRPC client used to report errors back to the
# query-frontend.
# The CLI flags prefix for this block configuration is:
# query-scheduler.grpc-client-config
[grpc_client_config: <grpc_client>]

# The maximum number of query-scheduler instances to use, regardless how many
# replicas are running. This option can be set only when
# -query-scheduler.service-discovery-mode is set to 'ring'. 0 to use all
# available query-scheduler instances.
# CLI flag: -query-scheduler.max-used-instances
[max_used_instances: <int> | default = 0]

store_gateway

The store_gateway block configures the store-gateway.

# The hash ring configuration.
sharding_ring:
  # The key-value store used to share the hash ring across multiple instances.
  kvstore:
    # Backend storage to use for the ring. Supported values are: consul, etcd,
    # inmemory, memberlist, multi.
    # CLI flag: -store-gateway.sharding-ring.store
    [store: <string> | default = "memberlist"]

    # The prefix for the keys in the store. Should end with a /.
    # CLI flag: -store-gateway.sharding-ring.prefix
    [prefix: <string> | default = "collectors/"]

    consul:
      # Hostname and port of Consul.
      # CLI flag: -store-gateway.sharding-ring.consul.hostname
      [host: <string> | default = "localhost:8500"]

      # ACL Token used to interact with Consul.
      # CLI flag: -store-gateway.sharding-ring.consul.acl-token
      [acl_token: <string> | default = ""]

      # HTTP timeout when talking to Consul
      # CLI flag: -store-gateway.sharding-ring.consul.client-timeout
      [http_client_timeout: <duration> | default = 20s]

      # Enable consistent reads to Consul.
      # CLI flag: -store-gateway.sharding-ring.consul.consistent-reads
      [consistent_reads: <boolean> | default = false]

      # Rate limit when watching key or prefix in Consul, in requests per
      # second. 0 disables the rate limit.
      # CLI flag: -store-gateway.sharding-ring.consul.watch-rate-limit
      [watch_rate_limit: <float> | default = 1]

      # Burst size used in rate limit. Values less than 1 are treated as 1.
      # CLI flag: -store-gateway.sharding-ring.consul.watch-burst-size
      [watch_burst_size: <int> | default = 1]

      # Maximum duration to wait before retrying a Compare And Swap (CAS)
      # operation.
      # CLI flag: -store-gateway.sharding-ring.consul.cas-retry-delay
      [cas_retry_delay: <duration> | default = 1s]

    etcd:
      # The etcd endpoints to connect to.
      # CLI flag: -store-gateway.sharding-ring.etcd.endpoints
      [endpoints: <list of strings> | default = []]

      # The dial timeout for the etcd connection.
      # CLI flag: -store-gateway.sharding-ring.etcd.dial-timeout
      [dial_timeout: <duration> | default = 10s]

      # The maximum number of retries to do for failed ops.
      # CLI flag: -store-gateway.sharding-ring.etcd.max-retries
      [max_retries: <int> | default = 10]

      # Enable TLS.
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-enabled
      [tls_enabled: <boolean> | default = false]

      # Path to the client certificate, which will be used for authenticating
      # with the server. Also requires the key path to be configured.
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-cert-path
      [tls_cert_path: <string> | default = ""]

      # Path to the key for the client certificate. Also requires the client
      # certificate to be configured.
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-key-path
      [tls_key_path: <string> | default = ""]

      # Path to the CA certificates to validate server certificate against. If
      # not set, the host's root CA certificates are used.
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-ca-path
      [tls_ca_path: <string> | default = ""]

      # Override the expected name on the server certificate.
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-server-name
      [tls_server_name: <string> | default = ""]

      # Skip validating server certificate.
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-insecure-skip-verify
      [tls_insecure_skip_verify: <boolean> | default = false]

      # Override the default cipher suite list (separated by commas). Allowed
      # values:
      # 
      # Secure Ciphers:
      # - TLS_AES_128_GCM_SHA256
      # - TLS_AES_256_GCM_SHA384
      # - TLS_CHACHA20_POLY1305_SHA256
      # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
      # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
      # - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      # - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      # - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
      # 
      # Insecure Ciphers:
      # - TLS_RSA_WITH_RC4_128_SHA
      # - TLS_RSA_WITH_3DES_EDE_CBC_SHA
      # - TLS_RSA_WITH_AES_128_CBC_SHA
      # - TLS_RSA_WITH_AES_256_CBC_SHA
      # - TLS_RSA_WITH_AES_128_CBC_SHA256
      # - TLS_RSA_WITH_AES_128_GCM_SHA256
      # - TLS_RSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
      # - TLS_ECDHE_RSA_WITH_RC4_128_SHA
      # - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
      # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-cipher-suites
      [tls_cipher_suites: <string> | default = ""]

      # Override the default minimum TLS version. Allowed values: VersionTLS10,
      # VersionTLS11, VersionTLS12, VersionTLS13
      # CLI flag: -store-gateway.sharding-ring.etcd.tls-min-version
      [tls_min_version: <string> | default = ""]

      # Etcd username.
      # CLI flag: -store-gateway.sharding-ring.etcd.username
      [username: <string> | default = ""]

      # Etcd password.
      # CLI flag: -store-gateway.sharding-ring.etcd.password
      [password: <string> | default = ""]

    multi:
      # Primary backend storage used by multi-client.
      # CLI flag: -store-gateway.sharding-ring.multi.primary
      [primary: <string> | default = ""]

      # Secondary backend storage used by multi-client.
      # CLI flag: -store-gateway.sharding-ring.multi.secondary
      [secondary: <string> | default = ""]

      # Mirror writes to secondary store.
      # CLI flag: -store-gateway.sharding-ring.multi.mirror-enabled
      [mirror_enabled: <boolean> | default = false]

      # Timeout for storing value to secondary store.
      # CLI flag: -store-gateway.sharding-ring.multi.mirror-timeout
      [mirror_timeout: <duration> | default = 2s]

  # Period at which to heartbeat to the ring. 0 = disabled.
  # CLI flag: -store-gateway.sharding-ring.heartbeat-period
  [heartbeat_period: <duration> | default = 15s]

  # The heartbeat timeout after which store-gateways are considered unhealthy
  # within the ring. 0 = never (timeout disabled).
  # CLI flag: -store-gateway.sharding-ring.heartbeat-timeout
  [heartbeat_timeout: <duration> | default = 1m]

  # Instance ID to register in the ring.
  # CLI flag: -store-gateway.sharding-ring.instance-id
  [instance_id: <string> | default = "<hostname>"]

  # List of network interface names to look up when finding the instance IP
  # address.
  # CLI flag: -store-gateway.sharding-ring.instance-interface-names
  [instance_interface_names: <list of strings> | default = [<private network interfaces>]]

  # Port to advertise in the ring (defaults to -server.http-listen-port).
  # CLI flag: -store-gateway.sharding-ring.instance-port
  [instance_port: <int> | default = 0]

  # IP address to advertise in the ring. Default is auto-detected.
  # CLI flag: -store-gateway.sharding-ring.instance-addr
  [instance_addr: <string> | default = ""]

  # Enable using a IPv6 instance address. (default false)
  # CLI flag: -store-gateway.sharding-ring.instance-enable-ipv6
  [instance_enable_ipv6: <boolean> | default = false]

  # The replication factor to use when sharding blocks. This option needs be set
  # both on the store-gateway and querier when running in microservices mode.
  # CLI flag: -store-gateway.sharding-ring.replication-factor
  [replication_factor: <int> | default = 1]

  # File path where tokens are stored. If empty, tokens are not stored at
  # shutdown and restored at startup.
  # CLI flag: -store-gateway.sharding-ring.tokens-file-path
  [tokens_file_path: <string> | default = ""]

  # True to enable zone-awareness and replicate blocks across different
  # availability zones. This option needs be set both on the store-gateway and
  # querier when running in microservices mode.
  # CLI flag: -store-gateway.sharding-ring.zone-awareness-enabled
  [zone_awareness_enabled: <boolean> | default = false]

  # Minimum time to wait for ring stability at startup, if set to positive
  # value.
  # CLI flag: -store-gateway.sharding-ring.wait-stability-min-duration
  [wait_stability_min_duration: <duration> | default = 0s]

  # Maximum time to wait for ring stability at startup. If the store-gateway
  # ring keeps changing after this period of time, the store-gateway will start
  # anyway.
  # CLI flag: -store-gateway.sharding-ring.wait-stability-max-duration
  [wait_stability_max_duration: <duration> | default = 5m]

  # The availability zone where this instance is running. Required if
  # zone-awareness is enabled.
  # CLI flag: -store-gateway.sharding-ring.instance-availability-zone
  [instance_availability_zone: <string> | default = ""]

  # Unregister from the ring upon clean shutdown.
  # CLI flag: -store-gateway.sharding-ring.unregister-on-shutdown
  [unregister_on_shutdown: <boolean> | default = true]

bucket_store:
  # Directory to store synchronized pyroscope block headers. This directory is
  # not required to be persisted between restarts, but it's highly recommended
  # in order to improve the store-gateway startup time.
  # CLI flag: -blocks-storage.bucket-store.sync-dir
  [sync_dir: <string> | default = "./data/pyroscope-sync/"]

  # How frequently to scan the bucket, or to refresh the bucket index (if
  # enabled), in order to look for changes (new blocks shipped by ingesters and
  # blocks deleted by retention or compaction).
  # CLI flag: -blocks-storage.bucket-store.sync-interval
  [sync_interval: <duration> | default = 15m]

  # Maximum number of concurrent tenants synching blocks.
  # CLI flag: -blocks-storage.bucket-store.tenant-sync-concurrency
  [tenant_sync_concurrency: <int> | default = 10]

  # Blocks with minimum time within this duration are ignored, and not loaded by
  # store-gateway. Useful when used together with -querier.query-store-after to
  # prevent loading young blocks, because there are usually many of them
  # (depending on number of ingesters) and they are not yet compacted. Negative
  # values or 0 disable the filter.
  # CLI flag: -blocks-storage.bucket-store.ignore-blocks-within
  [ignore_blocks_within: <duration> | default = 3h]

  # Number of Go routines to use when syncing block meta files from object
  # storage per tenant.
  # CLI flag: -blocks-storage.bucket-store.meta-sync-concurrency
  [meta_sync_concurrency: <int> | default = 20]

  # Duration after which the blocks marked for deletion will be filtered out
  # while fetching blocks. The idea of ignore-deletion-marks-delay is to ignore
  # blocks that are marked for deletion with some delay. This ensures store can
  # still serve blocks that are meant to be deleted but do not have a
  # replacement yet.
  # CLI flag: -blocks-storage.bucket-store.ignore-deletion-marks-delay
  [ignore_deletion_mark_delay: <duration> | default = 30m]

compactor

The compactor block configures the compactor.

# List of compaction time ranges.
# CLI flag: -compactor.block-ranges
[block_ranges: <list of durations> | default = 1h0m0s,2h0m0s,8h0m0s]

# Number of Go routines to use when downloading blocks for compaction and
# uploading resulting blocks.
# CLI flag: -compactor.block-sync-concurrency
[block_sync_concurrency: <int> | default = 8]

# Number of Go routines to use when syncing block meta files from the long term
# storage.
# CLI flag: -compactor.meta-sync-concurrency
[meta_sync_concurrency: <int> | default = 20]

# Directory to temporarily store blocks during compaction. This directory is not
# required to be persisted between restarts.
# CLI flag: -compactor.data-dir
[data_dir: <string> | default = "./data-compactor"]

# The frequency at which the compaction runs
# CLI flag: -compactor.compaction-interval
[compaction_interval: <duration> | default = 30m]

# How many times to retry a failed compaction within a single compaction run.
# CLI flag: -compactor.compaction-retries
[compaction_retries: <int> | default = 3]

# Max number of concurrent compactions running.
# CLI flag: -compactor.compaction-concurrency
[compaction_concurrency: <int> | default = 1]

# How long the compactor waits before compacting first-level blocks that are
# uploaded by the ingesters. This configuration option allows for the reduction
# of cases where the compactor begins to compact blocks before all ingesters
# have uploaded their blocks to the storage.
# CLI flag: -compactor.first-level-compaction-wait-period
[first_level_compaction_wait_period: <duration> | default = 25m]

# How frequently compactor should run blocks cleanup and maintenance, as well as
# update the bucket index.
# CLI flag: -compactor.cleanup-interval
[cleanup_interval: <duration> | default = 15m]

# Max number of tenants for which blocks cleanup and maintenance should run
# concurrently.
# CLI flag: -compactor.cleanup-concurrency
[cleanup_concurrency: <int> | default = 20]

# Time before a block marked for deletion is deleted from bucket. If not 0,
# blocks will be marked for deletion and compactor component will permanently
# delete blocks marked for deletion from the bucket. If 0, blocks will be
# deleted straight away. Note that deleting blocks immediately can cause query
# failures.
# CLI flag: -compactor.deletion-delay
[deletion_delay: <duration> | default = 12h]

[tenant_cleanup_delay: <duration> | default = ]

# Max time for starting compactions for a single tenant. After this time no new
# compactions for the tenant are started before next compaction cycle. This can
# help in multi-tenant environments to avoid single tenant using all compaction
# time, but also in single-tenant environments to force new discovery of blocks
# more often. 0 = disabled.
# CLI flag: -compactor.max-compaction-time
[max_compaction_time: <duration> | default = 1h]

# If enabled, will delete the bucket-index, markers and debug files in the
# tenant bucket when there are no blocks left in the index.
# CLI flag: -compactor.no-blocks-file-cleanup-enabled
[no_blocks_file_cleanup_enabled: <boolean> | default = false]

# If enabled, the compactor will downsample profiles in blocks at compaction
# level 3 and above. The original profiles are also kept.
# CLI flag: -compactor.downsampler-enabled
[downsampler_enabled: <boolean> | default = false]

# Number of goroutines opening blocks before compaction.
# CLI flag: -compactor.max-opening-blocks-concurrency
[max_opening_blocks_concurrency: <int> | default = 16]

# Comma separated list of tenants that can be compacted. If specified, only
# these tenants will be compacted by compactor, otherwise all tenants can be
# compacted. Subject to sharding.
# CLI flag: -compactor.enabled-tenants
[enabled_tenants: <string> | default = ""]

# Comma separated list of tenants that cannot be compacted by this compactor. If
# specified, and compactor would normally pick given tenant for compaction (via
# -compactor.enabled-tenants or sharding), it will be ignored instead.
# CLI flag: -compactor.disabled-tenants
[disabled_tenants: <string> | default = ""]

sharding_ring:
  # The key-value store used to share the hash ring across multiple instances.
  kvstore:
    # Backend storage to use for the ring. Supported values are: consul, etcd,
    # inmemory, memberlist, multi.
    # CLI flag: -compactor.ring.store
    [store: <string> | default = "memberlist"]

    # The prefix for the keys in the store. Should end with a /.
    # CLI flag: -compactor.ring.prefix
    [prefix: <string> | default = "collectors/"]

    consul:
      # Hostname and port of Consul.
      # CLI flag: -compactor.ring.consul.hostname
      [host: <string> | default = "localhost:8500"]

      # ACL Token used to interact with Consul.
      # CLI flag: -compactor.ring.consul.acl-token
      [acl_token: <string> | default = ""]

      # HTTP timeout when talking to Consul
      # CLI flag: -compactor.ring.consul.client-timeout
      [http_client_timeout: <duration> | default = 20s]

      # Enable consistent reads to Consul.
      # CLI flag: -compactor.ring.consul.consistent-reads
      [consistent_reads: <boolean> | default = false]

      # Rate limit when watching key or prefix in Consul, in requests per
      # second. 0 disables the rate limit.
      # CLI flag: -compactor.ring.consul.watch-rate-limit
      [watch_rate_limit: <float> | default = 1]

      # Burst size used in rate limit. Values less than 1 are treated as 1.
      # CLI flag: -compactor.ring.consul.watch-burst-size
      [watch_burst_size: <int> | default = 1]

      # Maximum duration to wait before retrying a Compare And Swap (CAS)
      # operation.
      # CLI flag: -compactor.ring.consul.cas-retry-delay
      [cas_retry_delay: <duration> | default = 1s]

    etcd:
      # The etcd endpoints to connect to.
      # CLI flag: -compactor.ring.etcd.endpoints
      [endpoints: <list of strings> | default = []]

      # The dial timeout for the etcd connection.
      # CLI flag: -compactor.ring.etcd.dial-timeout
      [dial_timeout: <duration> | default = 10s]

      # The maximum number of retries to do for failed ops.
      # CLI flag: -compactor.ring.etcd.max-retries
      [max_retries: <int> | default = 10]

      # Enable TLS.
      # CLI flag: -compactor.ring.etcd.tls-enabled
      [tls_enabled: <boolean> | default = false]

      # Path to the client certificate, which will be used for authenticating
      # with the server. Also requires the key path to be configured.
      # CLI flag: -compactor.ring.etcd.tls-cert-path
      [tls_cert_path: <string> | default = ""]

      # Path to the key for the client certificate. Also requires the client
      # certificate to be configured.
      # CLI flag: -compactor.ring.etcd.tls-key-path
      [tls_key_path: <string> | default = ""]

      # Path to the CA certificates to validate server certificate against. If
      # not set, the host's root CA certificates are used.
      # CLI flag: -compactor.ring.etcd.tls-ca-path
      [tls_ca_path: <string> | default = ""]

      # Override the expected name on the server certificate.
      # CLI flag: -compactor.ring.etcd.tls-server-name
      [tls_server_name: <string> | default = ""]

      # Skip validating server certificate.
      # CLI flag: -compactor.ring.etcd.tls-insecure-skip-verify
      [tls_insecure_skip_verify: <boolean> | default = false]

      # Override the default cipher suite list (separated by commas). Allowed
      # values:
      # 
      # Secure Ciphers:
      # - TLS_AES_128_GCM_SHA256
      # - TLS_AES_256_GCM_SHA384
      # - TLS_CHACHA20_POLY1305_SHA256
      # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
      # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
      # - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      # - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      # - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      # - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
      # 
      # Insecure Ciphers:
      # - TLS_RSA_WITH_RC4_128_SHA
      # - TLS_RSA_WITH_3DES_EDE_CBC_SHA
      # - TLS_RSA_WITH_AES_128_CBC_SHA
      # - TLS_RSA_WITH_AES_256_CBC_SHA
      # - TLS_RSA_WITH_AES_128_CBC_SHA256
      # - TLS_RSA_WITH_AES_128_GCM_SHA256
      # - TLS_RSA_WITH_AES_256_GCM_SHA384
      # - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
      # - TLS_ECDHE_RSA_WITH_RC4_128_SHA
      # - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
      # - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
      # - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
      # CLI flag: -compactor.ring.etcd.tls-cipher-suites
      [tls_cipher_suites: <string> | default = ""]

      # Override the default minimum TLS version. Allowed values: VersionTLS10,
      # VersionTLS11, VersionTLS12, VersionTLS13
      # CLI flag: -compactor.ring.etcd.tls-min-version
      [tls_min_version: <string> | default = ""]

      # Etcd username.
      # CLI flag: -compactor.ring.etcd.username
      [username: <string> | default = ""]

      # Etcd password.
      # CLI flag: -compactor.ring.etcd.password
      [password: <string> | default = ""]

    multi:
      # Primary backend storage used by multi-client.
      # CLI flag: -compactor.ring.multi.primary
      [primary: <string> | default = ""]

      # Secondary backend storage used by multi-client.
      # CLI flag: -compactor.ring.multi.secondary
      [secondary: <string> | default = ""]

      # Mirror writes to secondary store.
      # CLI flag: -compactor.ring.multi.mirror-enabled
      [mirror_enabled: <boolean> | default = false]

      # Timeout for storing value to secondary store.
      # CLI flag: -compactor.ring.multi.mirror-timeout
      [mirror_timeout: <duration> | default = 2s]

  # Period at which to heartbeat to the ring. 0 = disabled.
  # CLI flag: -compactor.ring.heartbeat-period
  [heartbeat_period: <duration> | default = 15s]

  # The heartbeat timeout after which compactors are considered unhealthy within
  # the ring. 0 = never (timeout disabled).
  # CLI flag: -compactor.ring.heartbeat-timeout
  [heartbeat_timeout: <duration> | default = 1m]

  # Instance ID to register in the ring.
  # CLI flag: -compactor.ring.instance-id
  [instance_id: <string> | default = "<hostname>"]

  # List of network interface names to look up when finding the instance IP
  # address.
  # CLI flag: -compactor.ring.instance-interface-names
  [instance_interface_names: <list of strings> | default = [<private network interfaces>]]

  # Port to advertise in the ring (defaults to -server.http-listen-port).
  # CLI flag: -compactor.ring.instance-port
  [instance_port: <int> | default = 0]

  # IP address to advertise in the ring. Default is auto-detected.
  # CLI flag: -compactor.ring.instance-addr
  [instance_addr: <string> | default = ""]

  # Enable using a IPv6 instance address. (default false)
  # CLI flag: -compactor.ring.instance-enable-ipv6
  [instance_enable_ipv6: <boolean> | default = false]

  # Minimum time to wait for ring stability at startup. 0 to disable.
  # CLI flag: -compactor.ring.wait-stability-min-duration
  [wait_stability_min_duration: <duration> | default = 0s]

  # Maximum time to wait for ring stability at startup. If the compactor ring
  # keeps changing after this period of time, the compactor will start anyway.
  # CLI flag: -compactor.ring.wait-stability-max-duration
  [wait_stability_max_duration: <duration> | default = 5m]

  # Timeout for waiting on compactor to become ACTIVE in the ring.
  # CLI flag: -compactor.ring.wait-active-instance-timeout
  [wait_active_instance_timeout: <duration> | default = 10m]

# The sorting to use when deciding which compaction jobs should run first for a
# given tenant. Supported values are: smallest-range-oldest-blocks-first,
# newest-blocks-first.
# CLI flag: -compactor.compaction-jobs-order
[compaction_jobs_order: <string> | default = "smallest-range-oldest-blocks-first"]

# Experimental: The strategy to use when splitting blocks during compaction.
# Supported values are: fingerprint, stacktracePartition.
# CLI flag: -compactor.compaction-split-by
[compaction_split_by: <string> | default = "fingerprint"]

grpc_client

The grpc_client block configures the gRPC client used to communicate between two Pyroscope components. The supported CLI flags <prefix> used to reference this configuration block are:

querier.frontend-client
query-frontend.grpc-client-config
query-scheduler.grpc-client-config

# gRPC client max receive message size (bytes).
# CLI flag: -<prefix>.grpc-max-recv-msg-size
[max_recv_msg_size: <int> | default = 104857600]

# gRPC client max send message size (bytes).
# CLI flag: -<prefix>.grpc-max-send-msg-size
[max_send_msg_size: <int> | default = 104857600]

# Use compression when sending messages. Supported values are: 'gzip', 'snappy'
# and '' (disable compression)
# CLI flag: -<prefix>.grpc-compression
[grpc_compression: <string> | default = ""]

# Rate limit for gRPC client; 0 means disabled.
# CLI flag: -<prefix>.grpc-client-rate-limit
[rate_limit: <float> | default = 0]

# Rate limit burst for gRPC client.
# CLI flag: -<prefix>.grpc-client-rate-limit-burst
[rate_limit_burst: <int> | default = 0]

# Enable backoff and retry when we hit rate limits.
# CLI flag: -<prefix>.backoff-on-ratelimits
[backoff_on_ratelimits: <boolean> | default = false]

backoff_config:
  # Minimum delay when backing off.
  # CLI flag: -<prefix>.backoff-min-period
  [min_period: <duration> | default = 100ms]

  # Maximum delay when backing off.
  # CLI flag: -<prefix>.backoff-max-period
  [max_period: <duration> | default = 10s]

  # Number of times to backoff and retry before failing.
  # CLI flag: -<prefix>.backoff-retries
  [max_retries: <int> | default = 10]

# Initial stream window size. Values less than the default are not supported and
# are ignored. Setting this to a value other than the default disables the BDP
# estimator.
# CLI flag: -<prefix>.initial-stream-window-size
[initial_stream_window_size: <int> | default = 63KiB1023B]

# Initial connection window size. Values less than the default are not supported
# and are ignored. Setting this to a value other than the default disables the
# BDP estimator.
# CLI flag: -<prefix>.initial-connection-window-size
[initial_connection_window_size: <int> | default = 63KiB1023B]

# Enable TLS in the gRPC client. This flag needs to be enabled when any other
# TLS flag is set. If set to false, insecure connection to gRPC server will be
# used.
# CLI flag: -<prefix>.tls-enabled
[tls_enabled: <boolean> | default = false]

# Path to the client certificate, which will be used for authenticating with the
# server. Also requires the key path to be configured.
# CLI flag: -<prefix>.tls-cert-path
[tls_cert_path: <string> | default = ""]

# Path to the key for the client certificate. Also requires the client
# certificate to be configured.
# CLI flag: -<prefix>.tls-key-path
[tls_key_path: <string> | default = ""]

# Path to the CA certificates to validate server certificate against. If not
# set, the host's root CA certificates are used.
# CLI flag: -<prefix>.tls-ca-path
[tls_ca_path: <string> | default = ""]

# Override the expected name on the server certificate.
# CLI flag: -<prefix>.tls-server-name
[tls_server_name: <string> | default = ""]

# Skip validating server certificate.
# CLI flag: -<prefix>.tls-insecure-skip-verify
[tls_insecure_skip_verify: <boolean> | default = false]

# Override the default cipher suite list (separated by commas). Allowed values:
# 
# Secure Ciphers:
# - TLS_AES_128_GCM_SHA256
# - TLS_AES_256_GCM_SHA384
# - TLS_CHACHA20_POLY1305_SHA256
# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
# - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
# - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
# - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
# - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
# - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
# - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
# 
# Insecure Ciphers:
# - TLS_RSA_WITH_RC4_128_SHA
# - TLS_RSA_WITH_3DES_EDE_CBC_SHA
# - TLS_RSA_WITH_AES_128_CBC_SHA
# - TLS_RSA_WITH_AES_256_CBC_SHA
# - TLS_RSA_WITH_AES_128_CBC_SHA256
# - TLS_RSA_WITH_AES_128_GCM_SHA256
# - TLS_RSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
# - TLS_ECDHE_RSA_WITH_RC4_128_SHA
# - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
# CLI flag: -<prefix>.tls-cipher-suites
[tls_cipher_suites: <string> | default = ""]

# Override the default minimum TLS version. Allowed values: VersionTLS10,
# VersionTLS11, VersionTLS12, VersionTLS13
# CLI flag: -<prefix>.tls-min-version
[tls_min_version: <string> | default = ""]

# The maximum amount of time to establish a connection. A value of 0 means
# default gRPC client connect timeout and backoff.
# CLI flag: -<prefix>.connect-timeout
[connect_timeout: <duration> | default = 5s]

# Initial backoff delay after first connection failure. Only relevant if
# ConnectTimeout > 0.
# CLI flag: -<prefix>.connect-backoff-base-delay
[connect_backoff_base_delay: <duration> | default = 1s]

# Maximum backoff delay when establishing a connection. Only relevant if
# ConnectTimeout > 0.
# CLI flag: -<prefix>.connect-backoff-max-delay
[connect_backoff_max_delay: <duration> | default = 5s]

cluster_validation:
  # Optionally define the cluster validation label.
  # CLI flag: -<prefix>.cluster-validation.label
  [label: <string> | default = ""]

memberlist

The memberlist block configures the Gossip memberlist.

# Name of the node in memberlist cluster. Defaults to hostname.
# CLI flag: -memberlist.nodename
[node_name: <string> | default = ""]

# Add random suffix to the node name.
# CLI flag: -memberlist.randomize-node-name
[randomize_node_name: <boolean> | default = true]

# The timeout for establishing a connection with a remote node, and for
# read/write operations.
# CLI flag: -memberlist.stream-timeout
[stream_timeout: <duration> | default = 2s]

# Multiplication factor used when sending out messages (factor * log(N+1)).
# CLI flag: -memberlist.retransmit-factor
[retransmit_factor: <int> | default = 4]

# How often to use pull/push sync.
# CLI flag: -memberlist.pullpush-interval
[pull_push_interval: <duration> | default = 30s]

# How often to gossip.
# CLI flag: -memberlist.gossip-interval
[gossip_interval: <duration> | default = 200ms]

# How many nodes to gossip to.
# CLI flag: -memberlist.gossip-nodes
[gossip_nodes: <int> | default = 3]

# How long to keep gossiping to dead nodes, to give them chance to refute their
# death.
# CLI flag: -memberlist.gossip-to-dead-nodes-time
[gossip_to_dead_nodes_time: <duration> | default = 30s]

# How soon can dead node's name be reclaimed with new address. 0 to disable.
# CLI flag: -memberlist.dead-node-reclaim-time
[dead_node_reclaim_time: <duration> | default = 0s]

# Enable message compression. This can be used to reduce bandwidth usage at the
# cost of slightly more CPU utilization.
# CLI flag: -memberlist.compression-enabled
[compression_enabled: <boolean> | default = true]

# How frequently to notify watchers when a key changes. Can reduce CPU activity
# in large memberlist deployments. 0 to notify without delay.
# CLI flag: -memberlist.notify-interval
[notify_interval: <duration> | default = 0s]

# Gossip address to advertise to other members in the cluster. Used for NAT
# traversal.
# CLI flag: -memberlist.advertise-addr
[advertise_addr: <string> | default = ""]

# Gossip port to advertise to other members in the cluster. Used for NAT
# traversal.
# CLI flag: -memberlist.advertise-port
[advertise_port: <int> | default = 7946]

# The cluster label is an optional string to include in outbound packets and
# gossip streams. Other members in the memberlist cluster will discard any
# message whose label doesn't match the configured one, unless the
# 'cluster-label-verification-disabled' configuration option is set to true.
# CLI flag: -memberlist.cluster-label
[cluster_label: <string> | default = ""]

# When true, memberlist doesn't verify that inbound packets and gossip streams
# have the cluster label matching the configured one. This verification should
# be disabled while rolling out the change to the configured cluster label in a
# live memberlist cluster.
# CLI flag: -memberlist.cluster-label-verification-disabled
[cluster_label_verification_disabled: <boolean> | default = false]

# Other cluster members to join. Can be specified multiple times. It can be an
# IP, hostname or an entry specified in the DNS Service Discovery format.
# CLI flag: -memberlist.join
[join_members: <list of strings> | default = []]

# Min backoff duration to join other cluster members.
# CLI flag: -memberlist.min-join-backoff
[min_join_backoff: <duration> | default = 1s]

# Max backoff duration to join other cluster members.
# CLI flag: -memberlist.max-join-backoff
[max_join_backoff: <duration> | default = 1m]

# Max number of retries to join other cluster members.
# CLI flag: -memberlist.max-join-retries
[max_join_retries: <int> | default = 10]

# Abort if this node fails the fast memberlist cluster joining procedure at
# startup. When enabled, it's guaranteed that other services, depending on
# memberlist, have an updated view over the cluster state when they're started.
# CLI flag: -memberlist.abort-if-fast-join-fails
[abort_if_cluster_fast_join_fails: <boolean> | default = false]

# Abort if this node fails to join memberlist cluster at startup. When enabled,
# it's not guaranteed that other services are started only after the cluster
# state has been successfully updated; use 'abort-if-fast-join-fails' instead.
# CLI flag: -memberlist.abort-if-join-fails
[abort_if_cluster_join_fails: <boolean> | default = false]

# If not 0, how often to rejoin the cluster. Occasional rejoin can help to fix
# the cluster split issue, and is harmless otherwise. For example when using
# only few components as a seed nodes (via -memberlist.join), then it's
# recommended to use rejoin. If -memberlist.join points to dynamic service that
# resolves to all gossiping nodes (eg. Kubernetes headless service), then rejoin
# is not needed.
# CLI flag: -memberlist.rejoin-interval
[rejoin_interval: <duration> | default = 0s]

# How long to keep LEFT ingesters in the ring.
# CLI flag: -memberlist.left-ingesters-timeout
[left_ingesters_timeout: <duration> | default = 5m]

# How long to keep obsolete entries in the KV store.
# CLI flag: -memberlist.obsolete-entries-timeout
[obsolete_entries_timeout: <duration> | default = 30s]

# Timeout for leaving memberlist cluster.
# CLI flag: -memberlist.leave-timeout
[leave_timeout: <duration> | default = 20s]

# Timeout for broadcasting all remaining locally-generated updates to other
# nodes when shutting down. Only used if there are nodes left in the memberlist
# cluster, and only applies to locally-generated updates, not to broadcast
# messages that are result of incoming gossip updates. 0 = no timeout, wait
# until all locally-generated updates are sent.
# CLI flag: -memberlist.broadcast-timeout-for-local-updates-on-shutdown
[broadcast_timeout_for_local_updates_on_shutdown: <duration> | default = 10s]

# How much space to use for keeping received and sent messages in memory for
# troubleshooting (two buffers). 0 to disable.
# CLI flag: -memberlist.message-history-buffer-bytes
[message_history_buffer_bytes: <int> | default = 0]

# Size of the buffered channel for the WatchPrefix function.
# CLI flag: -memberlist.watch-prefix-buffer-size
[watch_prefix_buffer_size: <int> | default = 128]

# IP address to listen on for gossip messages. Multiple addresses may be
# specified. Defaults to 0.0.0.0
# CLI flag: -memberlist.bind-addr
[bind_addr: <list of strings> | default = []]

# Port to listen on for gossip messages.
# CLI flag: -memberlist.bind-port
[bind_port: <int> | default = 7946]

# Timeout used when connecting to other nodes to send packet.
# CLI flag: -memberlist.packet-dial-timeout
[packet_dial_timeout: <duration> | default = 2s]

# Timeout for writing 'packet' data.
# CLI flag: -memberlist.packet-write-timeout
[packet_write_timeout: <duration> | default = 5s]

# Maximum number of concurrent writes to other nodes.
# CLI flag: -memberlist.max-concurrent-writes
[max_concurrent_writes: <int> | default = 3]

# Timeout for acquiring one of the concurrent write slots. After this time, the
# message will be dropped.
# CLI flag: -memberlist.acquire-writer-timeout
[acquire_writer_timeout: <duration> | default = 250ms]

# Enable TLS on the memberlist transport layer.
# CLI flag: -memberlist.tls-enabled
[tls_enabled: <boolean> | default = false]

# Path to the client certificate, which will be used for authenticating with the
# server. Also requires the key path to be configured.
# CLI flag: -memberlist.tls-cert-path
[tls_cert_path: <string> | default = ""]

# Path to the key for the client certificate. Also requires the client
# certificate to be configured.
# CLI flag: -memberlist.tls-key-path
[tls_key_path: <string> | default = ""]

# Path to the CA certificates to validate server certificate against. If not
# set, the host's root CA certificates are used.
# CLI flag: -memberlist.tls-ca-path
[tls_ca_path: <string> | default = ""]

# Override the expected name on the server certificate.
# CLI flag: -memberlist.tls-server-name
[tls_server_name: <string> | default = ""]

# Skip validating server certificate.
# CLI flag: -memberlist.tls-insecure-skip-verify
[tls_insecure_skip_verify: <boolean> | default = false]

# Override the default cipher suite list (separated by commas). Allowed values:
# 
# Secure Ciphers:
# - TLS_AES_128_GCM_SHA256
# - TLS_AES_256_GCM_SHA384
# - TLS_CHACHA20_POLY1305_SHA256
# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
# - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
# - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
# - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
# - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
# - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
# - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
# 
# Insecure Ciphers:
# - TLS_RSA_WITH_RC4_128_SHA
# - TLS_RSA_WITH_3DES_EDE_CBC_SHA
# - TLS_RSA_WITH_AES_128_CBC_SHA
# - TLS_RSA_WITH_AES_256_CBC_SHA
# - TLS_RSA_WITH_AES_128_CBC_SHA256
# - TLS_RSA_WITH_AES_128_GCM_SHA256
# - TLS_RSA_WITH_AES_256_GCM_SHA384
# - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
# - TLS_ECDHE_RSA_WITH_RC4_128_SHA
# - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
# CLI flag: -memberlist.tls-cipher-suites
[tls_cipher_suites: <string> | default = ""]

# Override the default minimum TLS version. Allowed values: VersionTLS10,
# VersionTLS11, VersionTLS12, VersionTLS13
# CLI flag: -memberlist.tls-min-version
[tls_min_version: <string> | default = ""]

limits

The limits block configures default and per-tenant limits imposed by components.

# Per-tenant ingestion rate limit in sample size per second. Units in MB.
# CLI flag: -distributor.ingestion-rate-limit-mb
[ingestion_rate_mb: <float> | default = 4]

# Per-tenant allowed ingestion burst size (in sample size). Units in MB. The
# burst size refers to the per-distributor local rate limiter, and should be set
# at least to the maximum profile size expected in a single push request.
# CLI flag: -distributor.ingestion-burst-size-mb
[ingestion_burst_size_mb: <float> | default = 2]

# Maximum length accepted for label names.
# CLI flag: -validation.max-length-label-name
[max_label_name_length: <int> | default = 1024]

# Maximum length accepted for label value. This setting also applies to the
# metric name.
# CLI flag: -validation.max-length-label-value
[max_label_value_length: <int> | default = 2048]

# Maximum number of label names per series.
# CLI flag: -validation.max-label-names-per-series
[max_label_names_per_series: <int> | default = 30]

# Maximum number of sessions per series. 0 to disable.
# CLI flag: -validation.max-sessions-per-series
[max_sessions_per_series: <int> | default = 0]

# Enforce labels order optimization.
# CLI flag: -validation.enforce-labels-order
[enforce_labels_order: <boolean> | default = false]

# Maximum size of a profile in bytes. This is based off the uncompressed size. 0
# to disable.
# CLI flag: -validation.max-profile-size-bytes
[max_profile_size_bytes: <int> | default = 4194304]

# Maximum number of samples in a profile. 0 to disable.
# CLI flag: -validation.max-profile-stacktrace-samples
[max_profile_stacktrace_samples: <int> | default = 16000]

# Maximum number of labels in a profile sample. 0 to disable.
# CLI flag: -validation.max-profile-stacktrace-sample-labels
[max_profile_stacktrace_sample_labels: <int> | default = 100]

# Maximum depth of a profile stacktrace. Profiles are not rejected instead
# stacktraces are truncated. 0 to disable.
# CLI flag: -validation.max-profile-stacktrace-depth
[max_profile_stacktrace_depth: <int> | default = 1000]

# Maximum length of a profile symbol value (labels, function names and
# filenames, etc...). Profiles are not rejected instead symbol values are
# truncated. 0 to disable.
# CLI flag: -validation.max-profile-symbol-value-length
[max_profile_symbol_value_length: <int> | default = 65535]

distributor_usage_groups:

# Duration of the distributor aggregation window. Requires aggregation period to
# be specified. 0 to disable.
# CLI flag: -distributor.aggregation-window
[distributor_aggregation_window: <duration> | default = 0s]

# Duration of the distributor aggregation period. Requires aggregation window to
# be specified. 0 to disable.
# CLI flag: -distributor.aggregation-period
[distributor_aggregation_period: <duration> | default = 0s]

# List of ingestion relabel configurations. The relabeling rules work the same
# way, as those of
# [Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).
# All rules are applied in the order they are specified. Note: In most
# situations, it is more effective to use relabeling directly in Grafana Alloy.
# Example:
#   This example consists of two rules, the first one will drop all profiles
#   received with an label 'environment="secrets"' and the second rule will add
#   a label 'powered_by="Grafana Labs"' to all profile series.
#   ingestion_relabeling_rules:
#       - action: drop
#         regex: secret
#         source_labels:
#           - environment
#       - action: replace
#         replacement: grafana-labs
#         target_label: powered_by
# CLI flag: -distributor.ingestion-relabeling-rules
[ingestion_relabeling_rules: <list of Configs> | default = []]

# Position of the default ingestion relabeling rules in relation to relabel
# rules from overrides. Valid values are 'first', 'last' or 'disabled'.
# CLI flag: -distributor.ingestion-relabeling-default-rules-position
[ingestion_relabeling_default_rules_position: <string> | default = "first"]

# List of sample type relabel configurations. Rules are applied to sample types
# with __type__ and __unit__ labels, along with all series labels.
# Example:
#   This example shows sample type filtering rules. The first rule drops all
#   allocation-related sample types (alloc_objects, alloc_space) from memory
#   profiles, keeping only in-use metrics. The second rule keeps only
#   CPU-related sample types by matching the __type__ label. The third rule
#   shows how to drop allocation sample types for a specific service by
#   combining __type__ and service_name labels.
#   sample_type_relabeling_rules:
#       - action: drop
#         regex: alloc_.*
#         source_labels:
#           - __type__
#       - action: keep
#         regex: cpu|wall
#         source_labels:
#           - __type__
#       - action: drop
#         regex: alloc_.*;my-service
#         separator: ;
#         source_labels:
#           - __type__
#           - service_name
# CLI flag: -distributor.sample-type-relabeling-rules
[sample_type_relabeling_rules: <list of Configs> | default = []]

# The tenant's shard size used by shuffle-sharding. Must be set both on
# ingesters and distributors. 0 disables shuffle sharding.
# CLI flag: -distributor.ingestion-tenant-shard-size
[ingestion_tenant_shard_size: <int> | default = 0]

# Maximum number of active series of profiles per tenant, per ingester. 0 to
# disable.
# CLI flag: -ingester.max-local-series-per-tenant
[max_local_series_per_tenant: <int> | default = 0]

# Maximum number of active series of profiles per tenant, across the cluster. 0
# to disable. When the global limit is enabled, each ingester is configured with
# a dynamic local limit based on the replication factor and the current number
# of healthy ingesters, and is kept updated whenever the number of ingesters
# change.
# CLI flag: -ingester.max-global-series-per-tenant
[max_global_series_per_tenant: <int> | default = 5000]

# Limit how far back in profiling data can be queried, up until lookback
# duration ago. This limit is enforced in the query frontend. If the requested
# time range is outside the allowed range, the request will not fail, but will
# be modified to only query data within the allowed time range. 0 to disable,
# default to 7d.
# CLI flag: -querier.max-query-lookback
[max_query_lookback: <duration> | default = 1w]

# The limit to length of queries. 0 to disable.
# CLI flag: -querier.max-query-length
[max_query_length: <duration> | default = 1d]

# Maximum number of queries that will be scheduled in parallel by the frontend.
# CLI flag: -querier.max-query-parallelism
[max_query_parallelism: <int> | default = 0]

# Whether query analysis is enabled in the query frontend. If disabled, the
# /AnalyzeQuery endpoint will return an empty response.
# CLI flag: -querier.query-analysis-enabled
[query_analysis_enabled: <boolean> | default = true]

# Whether the series portion of query analysis is enabled. If disabled, no
# series data (e.g., series count) will be calculated by the /AnalyzeQuery
# endpoint.
# CLI flag: -querier.query-analysis-series-enabled
[query_analysis_series_enabled: <boolean> | default = false]

# Maximum number of flame graph nodes by default. 0 to disable.
# CLI flag: -querier.max-flamegraph-nodes-default
[max_flamegraph_nodes_default: <int> | default = 8192]

# Maximum number of flame graph nodes allowed. 0 to disable.
# CLI flag: -querier.max-flamegraph-nodes-max
[max_flamegraph_nodes_max: <int> | default = 1048576]

# The tenant's shard size, used when store-gateway sharding is enabled. Value of
# 0 disables shuffle sharding for the tenant, that is all tenant blocks are
# sharded across all store-gateway replicas.
# CLI flag: -store-gateway.tenant-shard-size
[store_gateway_tenant_shard_size: <int> | default = 0]

# Split queries by a time interval and execute in parallel. The value 0 disables
# splitting by time
# CLI flag: -querier.split-queries-by-interval
[split_queries_by_interval: <duration> | default = 0s]

# Whether profiles should be sanitized when merging.
# CLI flag: -querier.sanitize-on-merge
[query_sanitize_on_merge: <boolean> | default = true]

# Delete blocks containing samples older than the specified retention period. 0
# to disable.
# CLI flag: -compactor.blocks-retention-period
[compactor_blocks_retention_period: <duration> | default = 0s]

# The number of shards to use when splitting blocks. 0 to disable splitting.
# CLI flag: -compactor.split-and-merge-shards
[compactor_split_and_merge_shards: <int> | default = 0]

# Number of stages split shards will be written to. Number of output split
# shards is controlled by -compactor.split-and-merge-shards.
# CLI flag: -compactor.split-and-merge-stage-size
[compactor_split_and_merge_stage_size: <int> | default = 0]

# Number of groups that blocks for splitting should be grouped into. Each group
# of blocks is then split separately. Number of output split shards is
# controlled by -compactor.split-and-merge-shards.
# CLI flag: -compactor.split-groups
[compactor_split_groups: <int> | default = 1]

# Max number of compactors that can compact blocks for single tenant. 0 to
# disable the limit and use all compactors.
# CLI flag: -compactor.compactor-tenant-shard-size
[compactor_tenant_shard_size: <int> | default = 0]

# If a partial block (unfinished block without meta.json file) hasn't been
# modified for this time, it will be marked for deletion. The minimum accepted
# value is 4h0m0s: a lower value will be ignored and the feature disabled. 0 to
# disable.
# CLI flag: -compactor.partial-block-deletion-delay
[compactor_partial_block_deletion_delay: <duration> | default = 1d]

# If enabled, the compactor will downsample profiles in blocks at compaction
# level 3 and above. The original profiles are also kept. Note: This set the
# default for the teanant overrides, in order to be effective it also requires
# compactor.downsampler-enabled to be set to true.
# CLI flag: -compactor.compactor-downsampler-enabled
[compactor_downsampler_enabled: <boolean> | default = true]

# S3 server-side encryption type. Required to enable server-side encryption
# overrides for a specific tenant. If not set, the default S3 client settings
# are used.
[s3_sse_type: <string> | default = ""]

# S3 server-side encryption KMS Key ID. Ignored if the SSE type override is not
# set.
[s3_sse_kms_key_id: <string> | default = ""]

# S3 server-side encryption KMS encryption context. If unset and the key ID
# override is set, the encryption context will not be provided to S3. Ignored if
# the SSE type override is not set.
[s3_sse_kms_encryption_context: <string> | default = ""]

# This limits how far into the past profiling data can be ingested. This limit
# is enforced in the distributor. 0 to disable, defaults to 1h.
# CLI flag: -validation.reject-older-than
[reject_older_than: <duration> | default = 1h]

# This limits how far into the future profiling data can be ingested. This limit
# is enforced in the distributor. 0 to disable, defaults to 10m.
# CLI flag: -validation.reject-newer-than
[reject_newer_than: <duration> | default = 10m]

s3_storage_backend

The s3_backend block configures the connection to Amazon S3 object storage backend.

# The S3 bucket endpoint. It could be an AWS S3 endpoint listed at
# https://docs.aws.amazon.com/general/latest/gr/s3.html or the address of an
# S3-compatible service in hostname:port format.
# CLI flag: -storage.s3.endpoint
[endpoint: <string> | default = ""]

# S3 region. If unset, the client will issue a S3 GetBucketLocation API call to
# autodetect it.
# CLI flag: -storage.s3.region
[region: <string> | default = ""]

# S3 bucket name
# CLI flag: -storage.s3.bucket-name
[bucket_name: <string> | default = ""]

# S3 secret access key
# CLI flag: -storage.s3.secret-access-key
[secret_access_key: <string> | default = ""]

# S3 access key ID
# CLI flag: -storage.s3.access-key-id
[access_key_id: <string> | default = ""]

# If enabled, use http:// for the S3 endpoint instead of https://. This could be
# useful in local dev/test environments while using an S3-compatible backend
# storage, like Minio.
# CLI flag: -storage.s3.insecure
[insecure: <boolean> | default = false]

# The signature version to use for authenticating against S3. Supported values
# are: v4, v2.
# CLI flag: -storage.s3.signature-version
[signature_version: <string> | default = "v4"]

# Deprecated, use s3.bucket-lookup-type instead. Set this to `true` to force the
# bucket lookup to be using path-style.
# CLI flag: -storage.s3.force-path-style
[force_path_style: <boolean> | default = false]

# S3 bucket lookup style, use one of: [path-style virtual-hosted-style auto]
# CLI flag: -storage.s3.bucket-lookup-type
[bucket_lookup_type: <string> | default = "auto"]

sse:
  # Enable AWS Server Side Encryption. Supported values: SSE-KMS, SSE-S3.
  # CLI flag: -storage.s3.sse.type
  [type: <string> | default = ""]

  # KMS Key ID used to encrypt objects in S3
  # CLI flag: -storage.s3.sse.kms-key-id
  [kms_key_id: <string> | default = ""]

  # KMS Encryption Context used for object encryption. It expects JSON formatted
  # string.
  # CLI flag: -storage.s3.sse.kms-encryption-context
  [kms_encryption_context: <string> | default = ""]

http:
  # The time an idle connection will remain idle before closing.
  # CLI flag: -storage.s3.http.idle-conn-timeout
  [idle_conn_timeout: <duration> | default = 1m30s]

  # The amount of time the client will wait for a servers response headers.
  # CLI flag: -storage.s3.http.response-header-timeout
  [response_header_timeout: <duration> | default = 2m]

  # If the client connects to S3 via HTTPS and this option is enabled, the
  # client will accept any certificate and hostname.
  # CLI flag: -storage.s3.http.insecure-skip-verify
  [insecure_skip_verify: <boolean> | default = false]

  # Maximum time to wait for a TLS handshake. 0 means no limit.
  # CLI flag: -storage.s3.tls-handshake-timeout
  [tls_handshake_timeout: <duration> | default = 10s]

  # The time to wait for a server's first response headers after fully writing
  # the request headers if the request has an Expect header. 0 to send the
  # request body immediately.
  # CLI flag: -storage.s3.expect-continue-timeout
  [expect_continue_timeout: <duration> | default = 1s]

  # Maximum number of idle (keep-alive) connections across all hosts. 0 means no
  # limit.
  # CLI flag: -storage.s3.max-idle-connections
  [max_idle_connections: <int> | default = 0]

  # Maximum number of idle (keep-alive) connections to keep per-host. If 0, a
  # built-in default value is used.
  # CLI flag: -storage.s3.max-idle-connections-per-host
  [max_idle_connections_per_host: <int> | default = 100]

  # Maximum number of connections per host. 0 means no limit.
  # CLI flag: -storage.s3.max-connections-per-host
  [max_connections_per_host: <int> | default = 0]

gcs_storage_backend

The gcs_backend block configures the connection to Google Cloud Storage object storage backend.

# GCS bucket name
# CLI flag: -storage.gcs.bucket-name
[bucket_name: <string> | default = ""]

# JSON either from a Google Developers Console client_credentials.json file, or
# a Google Developers service account key. Needs to be valid JSON, not a
# filesystem path. If empty, fallback to Google default logic:
# 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS
# environment variable. For workload identity federation, refer to
# https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation on
# how to generate the JSON configuration file for on-prem/non-Google cloud
# platforms.
# 2. A JSON file in a location known to the gcloud command-line tool:
# $HOME/.config/gcloud/application_default_credentials.json.
# 3. On Google Compute Engine it fetches credentials from the metadata server.
# CLI flag: -storage.gcs.service-account
[service_account: <string> | default = ""]

http:
  # The time an idle connection will remain idle before closing.
  # CLI flag: -storage.gcs.http.idle-conn-timeout
  [idle_conn_timeout: <duration> | default = 1m30s]

  # The amount of time the client will wait for a servers response headers.
  # CLI flag: -storage.gcs.http.response-header-timeout
  [response_header_timeout: <duration> | default = 2m]

  # If the client connects to GCS via HTTPS and this option is enabled, the
  # client will accept any certificate and hostname.
  # CLI flag: -storage.gcs.http.insecure-skip-verify
  [insecure_skip_verify: <boolean> | default = false]

  # Maximum time to wait for a TLS handshake. 0 means no limit.
  # CLI flag: -storage.gcs.tls-handshake-timeout
  [tls_handshake_timeout: <duration> | default = 10s]

  # The time to wait for a server's first response headers after fully writing
  # the request headers if the request has an Expect header. 0 to send the
  # request body immediately.
  # CLI flag: -storage.gcs.expect-continue-timeout
  [expect_continue_timeout: <duration> | default = 1s]

  # Maximum number of idle (keep-alive) connections across all hosts. 0 means no
  # limit.
  # CLI flag: -storage.gcs.max-idle-connections
  [max_idle_conns: <int> | default = 0]

  # Maximum number of idle (keep-alive) connections to keep per-host. If 0, a
  # built-in default value is used.
  # CLI flag: -storage.gcs.max-idle-connections-per-host
  [max_idle_conns_per_host: <int> | default = 100]

  # Maximum number of connections per host. 0 means no limit.
  # CLI flag: -storage.gcs.max-connections-per-host
  [max_conns_per_host: <int> | default = 0]

azure_storage_backend

The azure_storage_backend block configures the connection to Azure object storage backend.

# Azure Active Directory tenant ID. If set alongside `client-id` and
# `client-secret`, these values will be used for authentication via a client
# secret credential.
# CLI flag: -storage.azure.az-tenant-id
[az_tenant_id: <string> | default = ""]

# Azure Active Directory client ID. If set alongside `az-tenant-id` and
# `client-secret`, these values will be used for authentication via a client
# secret credential.
# CLI flag: -storage.azure.client-id
[client_id: <string> | default = ""]

# Azure Active Directory client secret. If set alongside `az-tenant-id` and
# `client-id`, these values will be used for authentication via a client secret
# credential.
# CLI flag: -storage.azure.client-secret
[client_secret: <string> | default = ""]

# Azure storage account name
# CLI flag: -storage.azure.account-name
[account_name: <string> | default = ""]

# Azure storage account key. If unset, Azure managed identities will be used for
# authentication instead.
# CLI flag: -storage.azure.account-key
[account_key: <string> | default = ""]

# If `connection-string` is set, the value of `endpoint-suffix` will not be
# used. Use this method over `account-key` if you need to authenticate via a SAS
# token. Or if you use the Azurite emulator.
# CLI flag: -storage.azure.connection-string
[connection_string: <string> | default = ""]

# Azure storage container name
# CLI flag: -storage.azure.container-name
[container_name: <string> | default = ""]

# Azure storage endpoint suffix without schema. The account name will be
# prefixed to this value to create the FQDN. If set to empty string, default
# endpoint suffix is used.
# CLI flag: -storage.azure.endpoint-suffix
[endpoint_suffix: <string> | default = ""]

# Number of retries for recoverable errors
# CLI flag: -storage.azure.max-retries
[max_retries: <int> | default = 3]

# User assigned managed identity. If empty, then System assigned identity is
# used.
# CLI flag: -storage.azure.user-assigned-id
[user_assigned_id: <string> | default = ""]

swift_storage_backend

The swift_storage_backend block configures the connection to OpenStack Object Storage (Swift) object storage backend.

# OpenStack Swift authentication API version. 0 to autodetect.
# CLI flag: -storage.swift.auth-version
[auth_version: <int> | default = 0]

# OpenStack Swift authentication URL
# CLI flag: -storage.swift.auth-url
[auth_url: <string> | default = ""]

# OpenStack Swift username.
# CLI flag: -storage.swift.username
[username: <string> | default = ""]

# OpenStack Swift user's domain name.
# CLI flag: -storage.swift.user-domain-name
[user_domain_name: <string> | default = ""]

# OpenStack Swift user's domain ID.
# CLI flag: -storage.swift.user-domain-id
[user_domain_id: <string> | default = ""]

# OpenStack Swift user ID.
# CLI flag: -storage.swift.user-id
[user_id: <string> | default = ""]

# OpenStack Swift API key.
# CLI flag: -storage.swift.password
[password: <string> | default = ""]

# OpenStack Swift user's domain ID.
# CLI flag: -storage.swift.domain-id
[domain_id: <string> | default = ""]

# OpenStack Swift user's domain name.
# CLI flag: -storage.swift.domain-name
[domain_name: <string> | default = ""]

# OpenStack Swift project ID (v2,v3 auth only).
# CLI flag: -storage.swift.project-id
[project_id: <string> | default = ""]

# OpenStack Swift project name (v2,v3 auth only).
# CLI flag: -storage.swift.project-name
[project_name: <string> | default = ""]

# ID of the OpenStack Swift project's domain (v3 auth only), only needed if it
# differs the from user domain.
# CLI flag: -storage.swift.project-domain-id
[project_domain_id: <string> | default = ""]

# Name of the OpenStack Swift project's domain (v3 auth only), only needed if it
# differs from the user domain.
# CLI flag: -storage.swift.project-domain-name
[project_domain_name: <string> | default = ""]

# OpenStack Swift Region to use (v2,v3 auth only).
# CLI flag: -storage.swift.region-name
[region_name: <string> | default = ""]

# Name of the OpenStack Swift container to put chunks in.
# CLI flag: -storage.swift.container-name
[container_name: <string> | default = ""]

# Max retries on requests error.
# CLI flag: -storage.swift.max-retries
[max_retries: <int> | default = 3]

# Time after which a connection attempt is aborted.
# CLI flag: -storage.swift.connect-timeout
[connect_timeout: <duration> | default = 10s]

# Time after which an idle request is aborted. The timeout watchdog is reset
# each time some data is received, so the timeout triggers after X time no data
# is received on a request.
# CLI flag: -storage.swift.request-timeout
[request_timeout: <duration> | default = 5s]

filesystem_storage_backend

The filesystem_storage_backend block configures the usage of local file system as object storage backend.

# Local filesystem storage directory.
# CLI flag: -storage.filesystem.dir
[dir: <string> | default = "./data-shared"]

analytics

The analytics block configures usage statistics collection. For more details about usage statistics, refer to Anonymous usage statistics reporting

# Enable anonymous usage statistics collection. For more details about usage
# statistics, refer to
# https://grafana.com/docs/pyroscope/latest/configure-server/anonymous-usage-statistics-reporting/
# CLI flag: -usage-stats.enabled
[reporting_enabled: <boolean> | default = true]

Configure storage

Wed, 08 Apr 2026 14:38:28 +0000

Configure storage

Grafana Pyroscope provides storage configuration options for local and object storage.

Configure disk storage
Learn about the ingester component and how to configure disk storage for Pyroscope.
Configure object storage backend
Learn how to configure Pyroscope to use different object storage backend implementations.

Configure Pyroscope memberlist

Wed, 08 Apr 2026 14:38:28 +0000

Configure Pyroscope memberlist

Hash rings are a distributed consistent hashing scheme and are widely used by Pyroscope for sharding and replication. Pyroscope only supports hash ring via the memberlist protocol. You can configure memberlist by either the CLI flag or its respective YAML config option.

Memberlist

Pyroscope uses memberlist as the KV store backend. At startup, a Pyroscope instance connects to other Pyroscope replicas to join the cluster. A Pyroscope instance discovers the other replicas to join by resolving the addresses configured in -memberlist.join. The -memberlist.join CLI flag must resolve to other replicas in the cluster and can be specified multiple times.

The -memberlist.join can be set to an address in the following formats:

<ip>:<port>
<hostname>:<port>
DNS service discovery

Note: At a minimum, configure one or more addresses that resolve to a consistent subset of replicas (for example, all the ingesters).

Note: If you’re running Pyroscope in Kubernetes, define a headless Kubernetes Service which resolves to the IP addresses of all Pyroscope pods, then set -memberlist.join to dnssrv+<service name>.<namespace>.svc.cluster.local:<port>.

Pyroscope supports TLS for memberlist connections between its components. To see all supported configuration parameters, refer to memberlist.

Configuring the memberlist address and port

By default, Pyroscope memberlist protocol listens on address 0.0.0.0 and port 7946. If you run multiple Pyroscope processes on the same node or the port 7946 is not available, you can change the bind and advertise port by setting the following parameters:

-memberlist.bind-addr: IP address to listen on the local machine.
-memberlist.bind-port: Port to listen on the local machine.
-memberlist.advertise-addr: IP address to advertise to other Pyroscope replicas. The other replicas will connect to this IP to talk to the instance.
-memberlist.advertise-port: Port to advertise to other Pyroscope replicas. The other replicas will connect to this port to talk to the instance.

Fine tuning memberlist changes propagation latency

The pyroscope_ring_oldest_member_timestamp metric can be used to measure the propagation of hash ring changes. This metric tracks the oldest heartbeat timestamp across all instances in the ring. You can execute the following query to measure the age of the oldest heartbeat timestamp in the ring:

max(time() - pyroscope_ring_oldest_member_timestamp{state="ACTIVE"})

The measured age shouldn’t be higher than the configured <prefix>.heartbeat-period plus a reasonable delta (for example, 15 seconds). If you experience a higher changes propagation latency, you can adjust the following settings:

Decrease -memberlist.gossip-interval
Increase -memberlist.gossip-nodes
Decrease -memberlist.pullpush-interval
Increase -memberlist.retransmit-factor

About Pyroscope DNS service discovery

Some clients in Pyroscope support service discovery via DNS to locate the addresses of backend servers to connect to. The following clients support service discovery via DNS:

Memberlist KV store
- -memberlist.join

Supported discovery modes

DNS service discovery supports different discovery modes. You select a discovery mode by adding one of the following supported prefixes to the address:

dns+
The domain name after the prefix is looked up as an A/AAAA query. For example: dns+memcached.local:11211.
dnssrv+
The domain name after the prefix is looked up as a SRV query, and then each SRV record is resolved as an A/AAAA record. For example: dnssrv+_memcached._tcp.memcached.namespace.svc.cluster.local.
dnssrvnoa+
The domain name after the prefix is looked up as a SRV query, with no A/AAAA lookup made after that. For example: dnssrvnoa+_memcached._tcp.memcached.namespace.svc.cluster.local.

If no prefix is provided, the provided IP or hostname is used without pre-resolving it.

Configure Pyroscope source code integration

Wed, 08 Apr 2026 14:38:28 +0000

Configuring source code integration

Currently, the Pyroscope source code integration only supports GitHub.

This guide walks you through setting up the Pyroscope source code integration with GitHub with minimal permissions for Grafana Pyroscope.

Creating a GitHub App

Go to your GitHub account settings
Navigate to Developer settings > GitHub Apps
Click New GitHub App
Configure the app with the following settings:

Basic Information
- GitHub App name: Choose a name for your app (e.g., “my-pyroscope”)
- Homepage URL: This is a required field, you can use any URL. (e.g., https://grafana.com/oss/pyroscope/)
- Callback URL: Set this to your Grafana installation URL with the GitHub callback path. (e.g., https://grafana.your-domain.com/a/grafana-pyroscope-app/github/callback)
Permissions

The GitHub App works without any extra permissions for public repositories. If you want to access private repositories, you need to add these permissions:
- Repository permissions:
  - Metadata: Read-only access
  - Contents: Read-only access
Where can this GitHub App be installed?
- Select Any account if you want to allow installation on any GitHub account
- Select Only on this account if you want to restrict installation to your account only
Click Create GitHub App
After creating the GitHub App, you should end up in the GitHub App settings, find the Client ID and take a note of it.
Now scroll down to the Client secrets section and click Generate a new client secret
Important: Copy the generated client secret immediately - you won’t be able to see it again after closing the dialog

For anything not covered in this guide, you can refer to the GitHub docs: Registering a GitHub App.

Configuring Pyroscope

This section explains how to configure the Pyroscope source code integration in Grafana Pyroscope. The integration requires three environment variables to be set:

Variable	Description	Required
`GITHUB_CLIENT_ID`	The Client ID of your GitHub App	Yes
`GITHUB_CLIENT_SECRET`	The Client Secret of your GitHub App	Yes
`GITHUB_SESSION_SECRET`	A random string used to encrypt the session	Yes

Using the Helm Chart

If you’re using the official Helm chart, follow these steps to configure the Pyroscope source code integration:

Create a Kubernetes secret containing the required values, this will also generate a new random session secret:

kubectl create secret generic pyroscope-github \
  "--from-literal=client_id=<The Client ID from the 6. step>" \
  "--from-literal=client_secret=<The Client secret from the 8. step>" \
  "--from-literal=session_secret=$(openssl rand -base64 48)"

Update your values.yaml to expose these secrets as environment variables:

pyroscope:
  extraEnvVars:
    GITHUB_CLIENT_ID:
      valueFrom:
        secretKeyRef:
          name: pyroscope-github
          key: client_id
    GITHUB_CLIENT_SECRET:
      valueFrom:
        secretKeyRef:
          name: pyroscope-github
          key: client_secret
    GITHUB_SESSION_SECRET:
      valueFrom:
        secretKeyRef:
          name: pyroscope-github
          key: session_secret

Apply the changes using helm upgrade:

Other Deployment Methods

For other deployment methods, ensure the same environment variables are set in your deployment configuration.

Verifying the integration is correctly configured

The configuration of the Pyroscope source code integration is now completed. In order to verify everything works as expected follow the user guide: Integrate your source code on GitHub with Pyroscope profiling data.

IP address logging of a reverse proxy

Wed, 08 Apr 2026 14:38:28 +0000

IP address logging of a reverse proxy

Note that using a reverse proxy in front of Grafana Pyroscope can make it difficult to troubleshoot errors. You can use the following settings to log the IP address passed along by the reverse proxy in headers such as X-Forwarded-For.

-server.log-source-ips-enabled

Set this to true to add IP address logging when a Forwarded, X-Real-IP, or X-Forwarded-For header is used. A field called sourceIPs is added to error logs when data is pushed into Grafana Pyroscope.
-server.log-source-ips-header

The header field stores the source IP addresses and is used only if -server.log-source-ips-enabled is true, and if -server.log-source-ips-regex is set. If you do not set these flags, the default Forwarded, X-Real-IP, or X-Forwarded-For headers are searched.
-server.log-source-ips-regex

A regular expression that is used to match the source IPs. The regular expression must contain at least one capturing group, the first of which is returned. This flag is used only if -server.log-source-ips-enabled is true and if -server.log-source-ips-header is set.

Configure Grafana Pyroscope shuffle sharding

Wed, 08 Apr 2026 14:38:28 +0000

Configure Grafana Pyroscope shuffle sharding

Grafana Pyroscope leverages sharding to horizontally scale both single- and multi-tenant clusters beyond the capacity of a single node.

Background

Grafana Pyroscope uses a sharding strategy that distributes the workload across a subset of the instances that run a given component. For example, on the write path, each tenant’s series are sharded across a subset of the ingesters. The size of this subset, which is the number of instances, is configured using the shard size parameter, which by default is 0. This default value means that each tenant uses all available instances, in order to fairly balance resources such as CPU and memory usage, and to maximize the usage of these resources across the cluster.

In a multi-tenant cluster this default (0) value introduces the following downsides:

An outage affects all tenants.
A misbehaving tenant, for example, a tenant that causes an out-of-memory error, can negatively affect all other tenants.

Configuring a shard size value higher than 0 enables shuffle sharding. The goal of shuffle sharding is to reduce the blast radius of an outage and better isolate tenants.

About shuffle sharding

Shuffle sharding is a technique that isolates different tenant’s workloads and gives each tenant a single-tenant experience, even if they’re running in a shared cluster. For more information about how AWS describes shuffle sharding, refer to What is shuffle sharding?.

Shuffle sharding assigns each tenant a shard that is composed of a subset of the Grafana Pyroscope instances. This technique minimizes the number of overlapping instances between two tenants. Shuffle sharding provides the following benefits:

An outage on some Grafana Pyroscope cluster instances or nodes only affect a subset of tenants.
A misbehaving tenant only affects its shard instances. Assuming that each tenant shard is relatively small compared to the total number of instances in the cluster, it’s likely that any other tenant runs on different instances or that only a subset of instances match the affected instances.

Using shuffle sharding doesn’t require more resources, but can result in unbalanced instances.

Low overlapping instances probability

For example, in a Grafana Pyroscope cluster that runs 50 ingesters and assigns each tenant four out of 50 ingesters, by shuffling instances between each tenant, there are 230,000 possible combinations.

Randomly picking two tenants yields the following probabilities:

71% chance that they do not share any instance
26% chance that they share only 1 instance
2.7% chance that they share 2 instances
0.08% chance that they share 3 instances
0.0004% chance that their instances fully overlap

Grafana Pyroscope shuffle sharding

Grafana Pyroscope supports shuffle sharding in the following components:

When you run Grafana Pyroscope with the default configuration, shuffle sharding is disabled and you need to explicitly enable it by increasing the shard size either globally or for a given tenant.

Note: If the shard size value is equal to or higher than the number of available instances, for example where -distributor.ingestion-tenant-shard-size is higher than the number of ingesters, then shuffle sharding is disabled and all instances are used again.

Guaranteed properties

The Grafana Pyroscope shuffle sharding implementation provides the following benefits:

Stability
Given a consistent state of the hash ring, the shuffle sharding algorithm always selects the same instances for a given tenant, even across different machines.
Consistency
Adding or removing an instance from the hash ring leads to, at most, only one instance changed in each tenant’s shard.
Shuffling
Probabilistically and for a large enough cluster, shuffle sharding ensures that every tenant receives a different set of instances with a reduced number of overlapping instances between two tenants, which improves failure isolation.

Ingesters shuffle sharding

By default, the Grafana Pyroscope distributor divides the received series among all running ingesters.

When you enable ingester shuffle sharding, the distributor on the write path divide each tenant series among -distributor.ingestion-tenant-shard-size number of ingesters, while on the read path, the querier queries only the subset of ingesters that hold the series for a given tenant.

The shard size can be overridden on a per-tenant basis by setting ingestion_tenant_shard_size in the overrides section of the runtime configuration.

Ingesters write path

To enable shuffle sharding for ingesters on the write path, configure the following flags (or their respective YAML configuration options) on the distributor, ingester, and ruler:

-distributor.ingestion-tenant-shard-size=<size>
<size>: Set the size to the number of ingesters each tenant series should be sharded to. If <size> is 0 or is greater than the number of available ingesters in the Grafana Pyroscope cluster, the tenant series are sharded across all ingesters.

Ingesters read path

Assuming that you have enabled shuffle sharding for the write path, to enable shuffle sharding for ingesters on the read path, configure the following flags (or their respective YAML configuration options) on the querier:

-distributor.ingestion-tenant-shard-size=<size>

The following flag is set appropriately by default to enable shuffle sharding for ingesters on the read path. If you need to modify its defaults:

-querier.shuffle-sharding-ingesters-enabled=true
Shuffle sharding for ingesters on the read path can be explicitly enabled or disabled.
- If shuffle sharding is enabled, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which might have received series since now - -blocks-storage.tsdb.retention-period. Otherwise, the request is sent to all ingesters.

If you enable ingesters shuffle sharding only for the write path, queriers on the read path always query all ingesters instead of querying the subset of ingesters that belong to the tenant’s shard. Keeping ingesters shuffle sharding enabled only on the write path does not lead to incorrect query results, but might increase query latency.

Rollout strategy

If you’re running a Grafana Pyroscope cluster with shuffle sharding disabled, and you want to enable it for the ingesters, use the following rollout strategy to avoid missing querying for any series currently in the ingesters:

Explicitly disable ingesters shuffle-sharding on the read path via -querier.shuffle-sharding-ingesters-enabled=false since this is enabled by default.
Enable ingesters shuffle sharding on the write path.
Enable ingesters shuffle-sharding on the read path via -querier.shuffle-sharding-ingesters-enabled=true.

Limitation: Decreasing the tenant shard size

The current shuffle sharding implementation in Grafana Pyroscope has a limitation that prevents you from safely decreasing the tenant shard size when you enable ingesters’ shuffle sharding on the read path.

If a tenant’s shard decreases in size, there is currently no way for the queriers to know how large the tenant shard was previously, and as a result, they potentially miss an ingester with data for that tenant. The blocks-storage.tsdb.retention-period, which is used to select the ingesters that might have received series since ’now - blocks-storage.tsdb.retention-period’, doesn’t work correctly for finding tenant shards if the tenant shard size is decreased.

Although decreasing the tenant shard size is not supported, consider the following workaround:

Disable shuffle sharding on the read path via -querier.shuffle-sharding-ingesters-enabled=false.
Decrease the configured tenant shard size.
Wait for at least the amount of time specified via -blocks-storage.tsdb.retention-period.
Re-enable shuffle sharding on the read path via -querier.shuffle-sharding-ingesters-enabled=true.

Query-frontend and query-scheduler shuffle sharding

By default, all Grafana Pyroscope queriers can execute queries for any tenant.

When you enable shuffle sharding by setting -query-frontend.max-queriers-per-tenant (or its respective YAML configuration option) to a value higher than 0 and lower than the number of available queriers, only the specified number of queriers are eligible to execute queries for a given tenant.

Note that this distribution happens in query-frontend, or query-scheduler, if used. When using query-scheduler, the -query-frontend.max-queriers-per-tenant option must be set for the query-scheduler component. When you don’t use query-frontend (with or without query-scheduler), this option is not available.

You can override the maximum number of queriers on a per-tenant basis by setting max_queriers_per_tenant in the overrides section of the runtime configuration.

The impact of a “query of death”

In the event a tenant sends a “query of death” which causes a querier to crash, the crashed querier becomes disconnected from the query-frontend or query-scheduler, and another running querier is immediately assigned to the tenant’s shard.

If the tenant repeatedly sends this query, the new querier assigned to the tenant’s shard crashes as well, and yet another querier is assigned to the shard. This cascading failure can potentially result in all running queriers to crash, one by one, which invalidates the assumption that shuffle sharding contains the blast radius of queries of death.

To mitigate this negative impact, there are experimental configuration options that enable you to configure a time delay between when a querier disconnects due to a crash and when the crashed querier is replaced by a healthy querier. When you configure a time delay, a tenant that repeatedly sends a “query of death” runs with reduced querier capacity after a querier has crashed. The tenant could end up having no available queriers, but this configuration reduces the likelihood that the crash impacts other tenants.

A delay of 1 minute might be a reasonable trade-off:

Query-frontend: -query-frontend.querier-forget-delay=1m
Query-scheduler: -query-scheduler.querier-forget-delay=1m

Store-gateway shuffle sharding

By default, a tenant’s blocks are divided among all Grafana Pyroscope store-gateways.

When you enable store-gateway shuffle sharding by setting -store-gateway.tenant-shard-size (or its respective YAML configuration option) to a value higher than 0 and lower than the number of available store-gateways, only the specified number of store-gateways are eligible to load and query blocks for a given tenant. You must set this flag on the store-gateway and querier.

You can override the store-gateway shard size on a per-tenant basis by setting store_gateway_tenant_shard_size in the overrides section of the runtime configuration.

For more information about the store-gateway, refer to store-gateway.

Compactor shuffle sharding

By default, tenant blocks can be compacted by any Grafana Pyroscope compactor.

When you enable compactor shuffle sharding by setting -compactor.compactor-tenant-shard-size (or its respective YAML configuration option) to a value higher than 0 and lower than the number of available compactors, only the specified number of compactors are eligible to compact blocks for a given tenant.

You can override the compactor shard size on a per-tenant basis setting by compactor_tenant_shard_size in the overrides section of the runtime configuration.

Shuffle sharding impact to the KV store

Shuffle sharding does not add additional overhead to the KV store. Shards are computed client-side and are not stored in the ring. KV store sizing depends primarily on the number of replicas of any component that uses the ring, for example, ingesters, and the number of tokens per replica.

However, in some components, each tenant’s shard is cached in-memory on the client-side, which might slightly increase their memory footprint. Increased memory footprint can happen mostly in the distributor.

Configure the Pyroscope server on Grafana Labs

About Grafana Pyroscope anonymous usage statistics reporting

About Grafana Pyroscope anonymous usage statistics reporting

The statistics server

Which information is collected

Disable the anonymous usage statistics reporting

About Pyroscope configurations

About Pyroscope configurations

Operational considerations

Tenant IDs

Tenant IDs

Restrictions

Pyroscope configuration parameters

Pyroscope configuration parameters

Generic placeholders

Use environment variables in the configuration

Configuration parameters

server

distributor

ingester

querier

query_frontend

frontend_worker

query_scheduler

store_gateway

compactor

grpc_client

memberlist

limits

s3_storage_backend

gcs_storage_backend

azure_storage_backend

swift_storage_backend

filesystem_storage_backend

analytics

Configure storage

Configure storage

Configure Pyroscope memberlist

Configure Pyroscope memberlist

Memberlist

Configuring the memberlist address and port

Fine tuning memberlist changes propagation latency

About Pyroscope DNS service discovery

Supported discovery modes

Configure Pyroscope source code integration

Configuring source code integration

Creating a GitHub App

Basic Information

Permissions

Where can this GitHub App be installed?

Configuring Pyroscope

Using the Helm Chart

Other Deployment Methods

Verifying the integration is correctly configured

IP address logging of a reverse proxy

IP address logging of a reverse proxy

Configure Grafana Pyroscope shuffle sharding

Configure Grafana Pyroscope shuffle sharding

Background

About shuffle sharding

Low overlapping instances probability

Grafana Pyroscope shuffle sharding

Guaranteed properties

Ingesters shuffle sharding

Ingesters write path

Ingesters read path

Rollout strategy

Limitation: Decreasing the tenant shard size

Query-frontend and query-scheduler shuffle sharding

The impact of a “query of death”

Store-gateway shuffle sharding

Compactor shuffle sharding

Shuffle sharding impact to the KV store