Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Open source

phlare.scrape

phlare.scrape configures a pprof scraping job for a given set of targets. The scraped performance profiles are forwarded to the list of receivers passed in forward_to.

Multiple phlare.scrape components can be specified by giving them different labels.

Usage

phlare.scrape "LABEL" {
  targets    = TARGET_LIST
  forward_to = RECEIVER_LIST
}

Arguments

The component configures and starts a new scrape job to scrape all of the input targets. Multiple scrape jobs can be spawned for a single input target when scraping multiple profile types.

The list of arguments that can be used to configure the block is presented below.

The scrape job name defaults to the component’s unique identifier.

Any omitted fields take on their default values. If conflicting attributes are being passed (e.g., defining both a BearerToken and BearerTokenFile or configuring both Basic Authorization and OAuth2 at the same time), the component reports an error.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
targetslist(map(string))List of targets to scrape.yes
forward_tolist(ProfilesReceiver)List of receivers to send scraped profiles to.yes
job_namestringThe job name to override the job label with.component nameno
paramsmap(list(string))A set of query parameters with which the target is scraped.no
scrape_intervaldurationHow frequently to scrape the targets of this scrape config."15s"no
scrape_timeoutdurationThe timeout for scraping targets of this config."15s"no
schemestringThe URL scheme with which to fetch metrics from targets.no

Blocks

The following blocks are supported inside the definition of phlare.scrape:

HierarchyBlockDescriptionRequired
http_client_confighttp_client_configHTTP client settings when connecting to targets.no
http_client_config > basic_authbasic_authConfigure basic_auth for authenticating to targets.no
http_client_config > authorizationauthorizationConfigure generic authorization to targets.no
http_client_config > oauth2oauth2Configure OAuth2 for authenticating to targets.no
http_client_config > oauth2 > tls_configtls_configConfigure TLS settings for connecting to targets via OAuth2.no
http_client_config > tls_configtls_configConfigure TLS settings for connecting to targets.no
profiling_configprofiling_configConfigure profiling settings for the scrape job.no
profiling_config > profile.memoryprofile.memoryCollect memory profiles.no
profiling_config > profile.blockprofile.blockCollect profiles on blocks.no
profiling_config > profile.goroutineprofile.goroutineCollect goroutine profiles.no
profiling_config > profile.mutexprofile.mutexCollect mutex profiles.no
profiling_config > profile.process_cpuprofile.process_cpuCollect CPU profiles.no
profiling_config > profile.fgprofprofile.fgprofCollect fgprof profiles.no
profiling_config > profile.customprofile.customCollect custom profiles.no

The > symbol indicates deeper levels of nesting. For example, http_client_config > basic_auth refers to a basic_auth block defined inside an http_client_config block.

http_client_config block

The http_client_config block configures settings used to connect to endpoints.

NameTypeDescriptionDefaultRequired
bearer_tokensecretBearer token to authenticate with.no
bearer_token_filestringFile containing a bearer token to authenticate with.no
proxy_urlstringHTTP proxy to proxy requests through.no
follow_redirectsboolWhether redirects returned by the server should be followed.trueno
enable_http_2boolWhether HTTP2 is supported for requests.trueno

bearer_token, bearer_token_file, basic_auth, authorization, and oauth2 are mutually exclusive and only one can be provided inside of a http_client_config block.

basic_auth block

NameTypeDescriptionDefaultRequired
usernamestringBasic auth username.no
passwordsecretBasic auth password.no
password_filestringFile containing the basic auth password.no

password and password_file are mututally exclusive and only one can be provided inside of a basic_auth block.

authorization block

NameTypeDescriptionDefaultRequired
typestringAuthorization type, for example, “Bearer”.no
credentialsecretSecret value.no
credentials_filestringFile containing the secret value.no

credential and credentials_file are mutually exclusive and only one can be provided inside of an authorization block.

oauth2 block

NameTypeDescriptionDefaultRequired
client_idstringOAuth2 client ID.no
client_secretsecretOAuth2 client secret.no
client_secret_filestringFile containing the OAuth2 client secret.no
scopeslist(string)List of scopes to authenticate with.no
token_urlstringURL to fetch the token from.no
endpoint_paramsmap(string)Optional parameters to append to the token URL.no
proxy_urlstringOptional proxy URL for OAuth2 requests.no

client_secret and client_secret_file are mututally exclusive and only one can be provided inside of an oauth2 block.

The oauth2 block may also contain its own separate tls_config sub-block.

tls_config block

NameTypeDescriptionDefaultRequired
ca_filestringCA certificate to validate the server with.no
cert_filestringCertificate file for client authentication.no
key_filestringKey file for client authentication.no
server_namestringServerName extension to indicate the name of the server.no
insecure_skip_verifyboolDisables validation of the server certificate.no
min_versionstringMinimum acceptable TLS version.no

When min_version is not provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If min_version is provided, it must be set to one of the following strings:

  • "TLS10" (TLS 1.0)
  • "TLS11" (TLS 1.1)
  • "TLS12" (TLS 1.2)
  • "TLS13" (TLS 1.3)

profiling_config block

The profiling_config block configures the profiling settings when scraping targets.

The block contains the following attributes:

NameTypeDescriptionDefaultRequired
path_prefixstringThe path prefix to use when scraping targets.no

profile.memory block

The profile.memory block collects profiles on memory consumption.

It accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.trueno
pathstringThe path to the profile type on the target."/debug/pprof/memory"no
deltabooleanWhether to scrape the profile as a delta.falseno

When the delta argument is true, a seconds query parameter is automatically added to requests.

profile.block block

The profile.block block collects profiles on process blocking.

It accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.trueno
pathstringThe path to the profile type on the target."/debug/pprof/block"no
deltabooleanWhether to scrape the profile as a delta.falseno

When the delta argument is true, a seconds query parameter is automatically added to requests.

profile.goroutine block

The profile.goroutine block collects profiles on the number of goroutines.

It accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.trueno
pathstringThe path to the profile type on the target."/debug/pprof/goroutine"no
deltabooleanWhether to scrape the profile as a delta.falseno

When the delta argument is true, a seconds query parameter is automatically added to requests.

profile.mutex block

The profile.mutex block collects profiles on mutexes.

It accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.trueno
pathstringThe path to the profile type on the target."/debug/pprof/mutex"no
deltabooleanWhether to scrape the profile as a delta.falseno

When the delta argument is true, a seconds query parameter is automatically added to requests.

profile.process_cpu block

The profile.process_cpu block collects profiles on CPU consumption for the process.

It accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.trueno
pathstringThe path to the profile type on the target."/debug/pprof/profile"no
deltabooleanWhether to scrape the profile as a delta.trueno

When the delta argument is true, a seconds query parameter is automatically added to requests.

profile.fgprof block

The profile.fgprof block collects profiles from an fgprof endpoint.

It accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.falseno
pathstringThe path to the profile type on the target."/debug/fgprof"no
deltabooleanWhether to scrape the profile as a delta.trueno

When the delta argument is true, a seconds query parameter is automatically added to requests.

profile.custom block

The profile.custom block allows for collecting profiles from custom endpoints. Blocks must be specified with a label:

river
profile.custom "PROFILE_TYPE" {
  enabled = true
  path    = "PROFILE_PATH"
}

Multiple profile.custom blocks can be specified. Labels assigned to profile.custom blocks must be unique across the component.

The profile.custom block accepts the following arguments:

NameTypeDescriptionDefaultRequired
enabledbooleanEnable this profile type to be scraped.yes
pathstringThe path to the profile type on the target.yes
deltabooleanWhether to scrape the profile as a delta.falseno

When the delta argument is true, a seconds query parameter is automatically added to requests.

Exported fields

phlare.scrape does not export any fields that can be referenced by other components.

Component health

phlare.scrape is only reported as unhealthy if given an invalid configuration.

Debug information

phlare.scrape reports the status of the last scrape for each configured scrape job on the component’s debug endpoint.

Debug metrics

  • phlare_fanout_latency (histogram): Write latency for sending to direct and indirect components.

Scraping behavior

The phlare.scrape component borrows the scraping behavior of Prometheus. Prometheus, and by extension, this component, uses a pull model for scraping profiles from a given set of targets. Each scrape target is defined as a set of key-value pairs called labels.

The set of targets can either be static, or dynamically provided periodically by a service discovery component such as discovery.kubernetes. The special label __address__ must always be present and corresponds to the <host>:<port> that is used for the scrape request.

By default, the scrape job tries to scrape all available targets’ /debug/pprof endpoints using HTTP, with a scrape interval of 15 seconds and scrape timeout of 15 seconds. The profile paths, protocol scheme, scrape interval and timeout, query parameters, as well as any other settings can be configured using the component’s arguments.

The scrape job expects profiles exposed by the endpoint to follow the pprof protobuf format. All profiles are then propagated to each receiver listed in the component’s forward_to argument.

Labels coming from targets, that start with a double underscore __ are treated as internal, and are removed prior to scraping.

The phlare.scrape component regards a scrape as successful if it responded with an HTTP 200 OK status code and returned a body of valid pprof profile.

If the scrape request fails, the component’s debug UI section contains more detailed information about the failure, the last successful scrape, as well as the labels last used for scraping.

The following labels are automatically injected to the scraped profiles and can help pin down a scrape target.

LabelDescription
jobThe configured job name that the target belongs to. Defaults to the fully formed component name.
instanceThe __address__ or <host>:<port> of the scrape target’s URL.

Example

The following example sets up the scrape job with certain attributes (profiling config, targets) and lets it scrape two local applications (the Agent itself and Phlare). The exposed profiles are sent over to the provided list of receivers, as defined by other components.

river
phlare.scrape "local" {
  targets    = [
    {"__address__" = "localhost:4100", "app"="phlare"},
    {"__address__" = "localhost:12345", "app"="agent"},
  ]
  forward_to = [phlare.write.local.receiver]
  profiling_config {
    profile.fgprof {
      enabled = true
    }
    profile.block {
      enabled = false
    }
    profile.mutex {
      enabled = false
    }
  }
}

Here are the the endpoints that are being scraped every 15 seconds:

http://localhost:4100/debug/pprof/allocs
http://localhost:4100/debug/pprof/goroutine
http://localhost:4100/debug/pprof/profile?seconds=14
http://localhost:4100/debug/fgprof?seconds=14
http://localhost:12345/debug/pprof/allocs
http://localhost:12345/debug/pprof/goroutine
http://localhost:12345/debug/pprof/profile?seconds=14
http://localhost:12345/debug/fgprof?seconds=14