Documentation Grafana Cloud Instrument and send data Grafana Alloy Reference Components pyroscope pyroscope.scrape

pyroscope.scrape

pyroscope.scrape collects pprof performance profiles for a given set of HTTP targets.

pyroscope.scrape mimics the scraping behavior of prometheus.scrape. Similarly to how Prometheus scrapes metrics via HTTP, pyroscope.scrape collects profiles via HTTP requests.

Unlike Prometheus, which usually only scrapes one /metrics endpoint per target, pyroscope.scrape may need to scrape multiple endpoints for the same target. This is because the component scrapes different profile types from different endpoints. For example, the component may scrape mutex profiles from a /debug/pprof/delta_mutex HTTP endpoint, and memory consumption from a /debug/pprof/allocs HTTP endpoint.

You can configure the profile paths, protocol scheme, scrape interval, scrape timeout, query parameters, and any other settings within pyroscope.scrape.

The pyroscope.scrape component regards a scrape as successful if it responded with an HTTP 200 OK status code and returned the body of a valid pprof profile.

If a scrape request fails, the debug UI for pyroscope.scrape shows:

• Detailed information about the failure.
• The time of the last successful scrape.
• The labels last used for scraping.

The component can forward the scraped performance profiles to components such as pyroscope.write through the forward_to argument.

You can specify multiple pyroscope.scrape components by giving them different labels.

Usage

Alloy Copy

pyroscope.scrape "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Arguments

pyroscope.scrape starts a scrape job to scrape all of the input targets. The component can start multiple scrape jobs for a single input target when it scrapes multiple profile types.

You can use the following arguments with pyroscope.scrape:

You can provide at most one of the following:

• authorization block
• basic_auth block
• bearer_token_file argument
• bearer_token argument
• oauth2 block

Any omitted arguments take on their default values. If you pass conflicting arguments, for example, configuring both bearer_token and bearer_token_file, then pyroscope.scrape fails to start and reports an error.

job_name

The job_name argument defaults to the component’s unique identifier.

For example, the job_name of pyroscope.scrape "local" { ... } is "pyroscope.scrape.local".

targets

You can define the list of targets statically, dynamically, or as a combination of both.

The following special labels can change the behavior of pyroscope.scrape:

• __address__ is the special label that must always be present and corresponds to the <host>:<port> that’s used for the scrape request.
• __name__ is the special label that indicates which profile type the component collects.
• __profile_path__ is the special label that holds the path to the profile endpoint on the target (for example, “/debug/pprof/allocs”).
• __profile_path_prefix__ is the special label that holds an optional prefix to prepend to the profile path (for example, "/mimir-prometheus").
• service_name is a required label that identifies the service you profile.

The component treats labels that start with a double underscore as internal and removes them before scraping.

Every profile carries a service_name label. You can set it on the target yourself. If you don’t, pyroscope.scrape adds the label and infers a value from the following sources, in this order:

1. __meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name Pod annotation.
2. __meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
3. __meta_docker_container_name
4. __meta_dockerswarm_container_label_service_name or __meta_dockerswarm_service_name

When you don’t set service_name and the component can’t infer a value, it uses unspecified.

The component injects the following labels into the scraped profiles so that you can link them to a scrape target:

Alloy only sets otel.scope.name and otel.scope.version if they aren’t already present on the scraped profile. otel.scope.version is only set when otel.scope.name matches the default value.

scrape_interval

The scrape_interval typically refers to the frequency with which Alloy collects performance profiles from the monitored targets. It represents the time interval between consecutive scrapes or data collection events. This parameter is important for controlling the trade-off between resource usage and the freshness of the collected data.

If scrape_interval is short:

• Advantages:
  • The component loses fewer profiles if the scraped application crashes.
• Disadvantages:
  • Greater consumption of CPU, memory, and network resources during scrapes and remote writes.
  • The backend database (Pyroscope) consumes more storage space.

If scrape_interval is long:

• Advantages:
  • Lower resource consumption.
• Disadvantages:
  • The component loses more profiles if the scraped application crashes.
  • If you set the delta argument to true, the batch size of each remote write to Pyroscope may be bigger. You may need to tune the Pyroscope database with higher limits.
  • If you set the delta argument to true, you run a larger risk of reaching the HTTP server timeouts of the scraped application.

For example, consider this situation:

• You configure pyroscope.scrape with a scrape_interval of "60s".
• The scraped application runs an HTTP server with a timeout of 30 seconds.
• Any scrape HTTP requests where you set the delta argument to true fail, because they attempt to run for 59 seconds.

Blocks

You can use the following blocks with pyroscope.scrape:

Any omitted blocks take on their default values. For example, if you don’t specify profile.mutex in the configuration, the component uses the defaults documented in profile.mutex.

authorization

basic_auth

clustering

When Alloy is using clustering, and you set enabled to true, this pyroscope.scrape component instance opts in to participating in the cluster to distribute scrape load between all cluster nodes.

Clustering causes the set of targets to be locally filtered down to a unique subset per node, where each node is roughly assigned the same number of targets. If the state of the cluster changes, such as when a node joins, the cluster recalculates the subset of targets to scrape per node.

When you enable clustering mode, all Alloy instances participating in the cluster must use the same configuration file and have access to the same service discovery APIs.

If Alloy is not running in clustered mode, this block is a no-op.

oauth2

profiling_config

The profiling_config block configures the profiling settings when scraping targets.

You can use the following arguments with the profiling_config block:

profile.block

The profile.block block collects profiles on process blocking.

You can use the following arguments with the profile.block block:

For more information about the delta argument, see the delta argument section.

profile.custom

The profile.custom block collects profiles from custom endpoints. You must give each block a label:

Alloy Copy

profile.custom "<PROFILE_TYPE>" {
  enabled = true
  path    = "<PROFILE_PATH>"
}

You can specify multiple profile.custom blocks. Labels assigned to profile.custom blocks must be unique across the component.

You can use the following arguments with the profile.custom block:

When the delta argument is true, a seconds query parameter is automatically added to requests. The seconds used is equal to scrape_interval - 1.

profile.fgprof

The profile.fgprof block collects profiles from an fgprof endpoint.

You can use the following arguments with the profile.fgprof block:

For more information about the delta argument, see the delta argument section.

profile.godeltaprof_block

The profile.godeltaprof_block block collects profiles from the godeltaprof block endpoint. The target computes the delta.

You can use the following arguments with the profile.godeltaprof_block block:

profile.godeltaprof_memory

The profile.godeltaprof_memory block collects profiles from the godeltaprof memory endpoint. The target computes the delta.

You can use the following arguments with the profile.godeltaprof_memory block:

profile.godeltaprof_mutex

The profile.godeltaprof_mutex block collects profiles from the godeltaprof mutex endpoint. The target computes the delta.

You can use the following arguments with the profile.godeltaprof_mutex block:

profile.goroutine

The profile.goroutine block collects profiles on the number of goroutines.

You can use the following arguments with the profile.goroutine block:

Refer to delta argument for more information about the delta argument.

profile.memory

The profile.memory block collects profiles on memory consumption.

You can use the following arguments with the profile.memory block:

Refer to delta argument for more information about the delta argument.

profile.mutex

The profile.mutex block collects profiles on mutexes.

You can use the following arguments with the profile.mutex block:

Refer to delta argument for more information about the delta argument.

profile.process_cpu

The profile.process_cpu block collects profiles on CPU consumption for the process.

You can use the following arguments with the profile.process_cpu block:

For more information about the delta argument, see the delta argument section.

tls_config

Common configuration

The following configuration applies across the profile types described in Blocks.

delta argument

When the delta argument is false, the pprof HTTP query is instantaneous.

When the delta argument is true:

• The pprof HTTP query runs for a certain amount of time.
• A seconds parameter is automatically added to the HTTP request.
• The default value for the seconds query parameter is scrape_interval - 1. If you set delta_profiling_duration, then seconds takes the same value as delta_profiling_duration. However, the delta_profiling_duration can’t be larger than scrape_interval. For example, if you set scrape_interval to "15s", then seconds defaults to 14s If you set delta_profiling_duration to 16s, then you must set scrape_interval to at least 17s. If the HTTP endpoint is /debug/pprof/profile, then the HTTP query becomes /debug/pprof/profile?seconds=14

Exported fields

pyroscope.scrape doesn’t export any fields.

Component health

pyroscope.scrape is only reported as unhealthy if given an invalid configuration. In those cases, exported fields retain their last healthy values.

Debug information

pyroscope.scrape reports the status of the last scrape for each configured scrape job on the component’s debug endpoint.

Debug metrics

The following Prometheus metrics are exposed:

Examples

The following examples show how to scrape profiles from static and dynamically discovered targets, and how to enable and disable specific profile types.

Default endpoints of static targets

The following example sets up a scrape job of a statically configured list of targets - Alloy itself and Pyroscope. The component sends the scraped profiles to pyroscope.write, which remote writes them to a Pyroscope database.

Alloy Copy

pyroscope.scrape "local" {
  targets = [
    {"__address__" = "localhost:4040", "service_name"="pyroscope"},
    {"__address__" = "localhost:12345", "service_name"="alloy"},
  ]

  forward_to = [pyroscope.write.local.receiver]
}

pyroscope.write "local" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

The component scrapes these endpoints every 15 seconds:

text Copy

http://localhost:4040/debug/pprof/allocs
http://localhost:4040/debug/pprof/block
http://localhost:4040/debug/pprof/goroutine
http://localhost:4040/debug/pprof/mutex
http://localhost:4040/debug/pprof/profile?seconds=14

http://localhost:12345/debug/pprof/allocs
http://localhost:12345/debug/pprof/block
http://localhost:12345/debug/pprof/goroutine
http://localhost:12345/debug/pprof/mutex
http://localhost:12345/debug/pprof/profile?seconds=14

The component adds seconds=14 to the /debug/pprof/profile endpoint, because:

• The delta argument of the profile.process_cpu block is true by default.
• scrape_interval is "15s" by default.

The component doesn’t scrape the /debug/fgprof endpoint, because the enabled argument of the profile.fgprof block is false by default.

Default endpoints of dynamic targets

Alloy Copy

discovery.http "dynamic_targets" {
  url = "https://example.com/scrape_targets"
  refresh_interval = "15s"
}

pyroscope.scrape "local" {
  targets = [discovery.http.dynamic_targets.targets]

  forward_to = [pyroscope.write.local.receiver]
}

pyroscope.write "local" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Default endpoints of static and dynamic targets

Alloy Copy

discovery.http "dynamic_targets" {
  url = "https://example.com/scrape_targets"
  refresh_interval = "15s"
}

pyroscope.scrape "local" {
  targets = array.concat([
    {"__address__" = "localhost:4040", "service_name"="pyroscope"},
    {"__address__" = "localhost:12345", "service_name"="alloy"},
  ], discovery.http.dynamic_targets.targets)

  forward_to = [pyroscope.write.local.receiver]
}

pyroscope.write "local" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Enable and disable profiles

Alloy Copy

pyroscope.scrape "local" {
  targets = [
    {"__address__" = "localhost:12345", "service_name"="alloy"},
  ]

  profiling_config {
    profile.fgprof {
      enabled = true
    }
    profile.block {
      enabled = false
    }
    profile.mutex {
      enabled = false
    }
  }

  forward_to = [pyroscope.write.local.receiver]
}

The component scrapes these endpoints every 15 seconds:

text Copy

http://localhost:12345/debug/pprof/allocs
http://localhost:12345/debug/pprof/goroutine
http://localhost:12345/debug/pprof/profile?seconds=14
http://localhost:12345/debug/fgprof?seconds=14

These endpoints are NOT scraped because they’re explicitly disabled:

text Copy

http://localhost:12345/debug/pprof/block
http://localhost:12345/debug/pprof/mutex

Compatible components

pyroscope.scrape can accept arguments from the following components:

• Components that export Targets
• Components that export Pyroscope ProfilesReceiver