pyroscope on Grafana Labs

pyroscope.ebpf

Mon, 30 Mar 2026 15:47:22 +0000

`pyroscope.ebpf`

pyroscope.ebpf configures an eBPF profiling job for the current host. The collected performance profiles are forwarded to the list of receivers passed in forward_to.

The pyroscope.ebpf component embeds the grafana/opentelemetry-ebpf-profiler which is a fork of open-telemetry/opentelemetry-ebpf-profiler.

Note
To use the pyroscope.ebpf component you must run Alloy as root and inside the host PID namespace.

Note
The profiler requires file system storage at /tmp/symb-cache to store symbol cache data. Ensure this directory is accessible and has sufficient storage space.

You can specify multiple pyroscope.ebpf components by giving them different labels, however it’s not recommended as it can lead to additional memory and CPU usage.

Supported languages

Native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host)
Broad set of HLLs (Hotspot JVM, Python, Ruby, PHP, Node.JS, V8, Perl).

Usage

pyroscope.ebpf "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Arguments

The component configures and starts a new eBPF profiling job to collect performance profiles from the current host.

You can use the following arguments with pyroscope.ebpf:

Name	Type	Description	Default	Required
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send collected profiles to.		yes
`targets`	`list(map(string))`	List of targets to group profiles by container ID.		yes
`build_id_cache_size`	`int`	Deprecated (no-op), previously controlled the size of the elf file build id -> symbols table LRU cache.	`64`	no
`cache_rounds`	`int`	Deprecated (no-op), previously controlled the number of cache rounds.		no
`collect_interval`	`duration`	How frequently to collect profiles.	`"15s"`	no
`collect_kernel_profile`	`bool`	Deprecated (no-op), previously enabled collection of kernelspace profiles.	`true`	no
`collect_user_profile`	`bool`	Deprecated (no-op), previously enabled collection of userspace profiles.	`true`	no
`container_id_cache_size`	`int`	The size of the PID -> container ID table LRU cache.	`1024`	no
`demangle`	`string`	C++ demangle mode. Available options are: `none`, `simplified`, `templates`, or `full`.	`"none"`	no
`dotnet_enabled`	`bool`	A flag to enable or disable .NET profiling.	`true`	no
`go_table_fallback`	`bool`	Deprecated (no-op), previously enabled symbol lookup in `.sym` / `.dynsym` sections when `.gopclntab` lookup failed.	`false`	no
`hotspot_enabled`	`bool`	A flag to enable ordisable hotspot profiling.	`true`	no
`perl_enabled`	`bool`	A flag to enable or disable Perl profiling.	`true`	no
`php_enabled`	`bool`	A flag to enable or disable PHP profiling.	`true`	no
`pid_cache_size`	`int`	Deprecated (no-op), previously controlled the size of the PID -> proc symbols table LRU cache.	`32`	no
`pid_map_size`	`int`	Deprecated (no-op), previously controlled the size of eBPF PID map.	`2048`	no
`python_enabled`	`bool`	A flag to enable or disable python profiling.	`true`	no
`ruby_enabled`	`bool`	A flag to enable or disable Ruby profiling.	`true`	no
`same_file_cache_size`	`int`	Deprecated (no-op), previously controlled the size of the elf file -> symbols table LRU cache.	`8`	no
`sample_rate`	`int`	How many times per second to collect profile samples.	`19`	no
`symbols_map_size`	`int`	Deprecated (no-op), previously controlled the size of eBPF symbols map .	`16384`	no
`load_probe`	`bool`	Enable loading uprobe dynamically during runtime.	`false`	no
`u_probe_links`	`list(string)`	List of user-space symbols to collect stack-trace from, e.g. `["/usr/lib/libc.so.6:malloc"]`.		no
`v8_enabled`	`bool`	A flag to enable/disable V8 profiling.	`true`	no
`off_cpu_threshold`	`float`	A flag to adjust the off-cpu profiling threshold between 0 and 1 as float.	`0`	no
`verbose_mode`	`bool`	Enable verbose logging for the eBPF profiler.	`false`	no
`lazy_mode`	`bool`	Enable lazy mode to defer eBPF profiler startup until targets are discovered.	`false`	no

Only the forward_to and targets fields are required. Omitted fields take their default values.

Several arguments are marked as “Deprecated (no-op)”. These arguments were previously used for configuring various cache sizes and behaviors, but they no longer have any effect. They are kept for backward compatibility but will be removed in a future release. It is recommended to remove these arguments from your configuration.

Blocks

pyroscope.ebpf doesn’t support any blocks.

Exported fields

pyroscope.ebpf doesn’t export any fields that can be referenced by other components.

Component health

pyroscope.ebpf is only reported as unhealthy if given an invalid configuration.

Debug information

elf_cache per build id and per same file symbol tables and their sizes in symbols count.
pid_cache per process elf symbol tables and their sizes in symbols count.
targets currently tracked active targets.

Debug metrics

pyroscope_ebpf_active_targets (gauge): Number of active targets the component tracks.
pyroscope_ebpf_pprofs_total (counter): Number of pprof profiles collected by the eBPF component.
pyroscope_ebpf_profiling_sessions_failing_total (counter): Number of profiling sessions failed.
pyroscope_ebpf_profiling_sessions_total (counter): Number of profiling sessions completed.
pyroscope_fanout_latency (histogram): Write latency for sending to direct and indirect components.

eBPF profiler internal metrics

The component also exposes internal metrics from the embedded eBPF profiler. These metrics are only emitted after they have been recorded at least once, so not all metrics appear on every host. The metrics carry an otel_scope_name="pyroscope.ebpf" label.

Notable metrics include:

Native unwinding

UnwindNativeAttempts_total (counter): Unwind attempts since the previous check.
UnwindNativeFrames_total (counter): Unwound frames since the previous check.
UnwindNativeStackDeltaStop_total (counter): Number of stop stack deltas in the native unwinder (success).
UnwindNativeSmallPC_total (counter): Number of times PC held a value smaller than 0x1000.
UnwindErrStackLengthExceeded_total (counter): Number of times MAX_FRAME_UNWINDS has been exceeded.

Interpreter unwinding

UnwindPythonAttempts_total (counter): Number of attempted Python unwinds.
UnwindPythonFrames_total (counter): Number of unwound Python frames.
UnwindHotspotAttempts_total (counter): Number of attempted Hotspot JVM unwinds.
UnwindHotspotFrames_total (counter): Number of unwound Hotspot JVM frames.
UnwindRubyAttempts_total (counter): Number of attempted Ruby unwinds.
UnwindRubyFrames_total (counter): Number of unwound Ruby frames.
UnwindPHPAttempts_total (counter): Number of attempted PHP unwinds.
UnwindPHPFrames_total (counter): Number of unwound PHP frames.
UnwindPerlAttempts_total (counter): Number of attempted Perl unwinds.
UnwindPerlFrames_total (counter): Number of unwound Perl frames.
UnwindV8Attempts_total (counter): Number of attempted V8 unwinds.
UnwindV8Frames_total (counter): Number of unwound V8 frames.
UnwindDotnetAttempts_total (counter): Number of attempted .NET unwinds.
UnwindDotnetFrames_total (counter): Number of unwound .NET frames.

Symbolization

PythonSymbolizationSuccesses_total (counter): Number of successfully symbolized Python frames.
PythonSymbolizationFailures_total (counter): Number of Python frames that failed symbolization.
HotspotSymbolizationSuccesses_total (counter): Number of successfully symbolized Hotspot frames.
HotspotSymbolizationFailures_total (counter): Number of Hotspot frames that failed symbolization.
RubySymbolizationSuccess_total (counter): Number of successfully symbolized Ruby frames.
RubySymbolizationFailure_total (counter): Number of Ruby frames that failed symbolization.

Process management

NumProcNew_total (counter): Number of new PID events.
NumProcExit_total (counter): Number of exit PID events.
NumGenericPID_total (counter): Number of generic PID events.

eBPF map state

NumExeIDLoadedToEBPF (gauge): The number of executables loaded to eBPF maps.
HashmapPidPageToMappingInfo (gauge): Current size of the pid_page_to_mapping_info hash map.
HashmapNumStackDeltaPages (gauge): Current size of the stack delta pages hash map.
UnwindInfoArraySize (gauge): Current size of the unwind info array.

The full list of ~213 metrics is defined in the opentelemetry-ebpf-profiler metrics.json.

Profile collecting behavior

The pyroscope.ebpf component collects stack traces associated with a process running on the current host. You can use the sample_rate argument to define the number of stack traces collected per second. The default is 97.

The following labels are automatically injected into the collected profiles if you haven’t defined them. These labels can help you pin down a profiling target.

Label	Description
`__container_id__`	The container ID derived from target.
`__name__`	Pyroscope metric name. Defaults to `process_cpu`.
`service_name`	Pyroscope service name. It’s automatically selected from discovery meta labels if possible. Otherwise defaults to `unspecified`.

Targets

One of the following special labels must be included in each target of targets and the label must correspond to the container or process that is profiled:

__container_id__: The container ID.
__meta_docker_container_id: The ID of the Docker container.
__meta_kubernetes_pod_container_id: The ID of the Kubernetes Pod container.
__process_pid__ : The process ID.

Each process is then associated with a specified target from the targets list, determined by a container ID or process PID.

If a process’s container ID matches a target’s container ID label, the stack traces are aggregated per target based on the container ID. If a process’s PID matches a target’s process PID label, the stack traces are aggregated per target based on the process PID. Otherwise the process isn’t profiled.

Service name

The special label service_name is required and must always be present. If it’s not specified, it’s attempted to be inferred from multiple sources:

__meta_docker_container_name
__meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
__meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name Pod annotation.

If service_name isn’t specified and couldn’t be inferred, it’s set to unspecified.

Troubleshoot unknown symbols

Symbols are extracted from various sources, including:

The .gopclntab section in Go language ELF files.
The .symtab and .dynsym sections in the debug ELF file.
The .symtab and .dynsym sections in the ELF file.

The search for debug files follows gdb algorithm. For example, if the profiler wants to find the debug file for /lib/x86_64-linux-gnu/libc.so.6 with a .gnu_debuglink set to libc.so.6.debug and a build ID 0123456789abcdef. The following paths are examined:

/usr/lib/debug/.build-id/01/0123456789abcdef.debug
/lib/x86_64-linux-gnu/libc.so.6.debug
/lib/x86_64-linux-gnu/.debug/libc.so.6.debug
/usr/lib/debug/lib/x86_64-linux-gnu/libc.so.6.debug

Deal with unknown symbols

Unknown symbols in the profiles you’ve collected indicate that the profiler couldn’t access an ELF file associated with a given address in the trace.

This can occur for several reasons:

The process has terminated, making the ELF file inaccessible.
The ELF file is either corrupted or not recognized as an ELF file.
There is no corresponding ELF file entry in /proc/pid/maps for the address in the stack trace.

Address unresolved symbols

If you only see module names without corresponding function names, for example, /lib/x86_64-linux-gnu/libc.so.6, it indicates that the symbols couldn’t be mapped to their respective function names.

This can occur for several reasons:

The binary has been stripped, leaving no .symtab, .dynsym, or .gopclntab sections in the ELF file.
The debug file is missing or couldn’t be located.

To fix this for your binaries, ensure that they’re either not stripped or that you have separate debug files available. You can achieve this by running:

objcopy --only-keep-debug elf elf.debug
strip elf -o elf.stripped
objcopy --add-gnu-debuglink=elf.debug elf.stripped elf.debuglink

For system libraries, ensure that debug symbols are installed. On Ubuntu, for example, you can install debug symbols for libc by executing:

apt install libc6-dbg

Understand flat stack traces

If your profiles show many shallow stack traces, typically 1-2 frames deep, your binary might have been compiled without frame pointers.

To compile your code with frame pointers, include the -fno-omit-frame-pointer flag in your compiler options.

Example

Kubernetes discovery

In the following example, performance profiles are collected from Pods on the same node, discovered using discovery.kubernetes. Pod selection relies on the HOSTNAME environment variable, which is a Pod name if Alloy is used as an Alloy Helm chart. The service_name label is set to {__meta_kubernetes_namespace}/{__meta_kubernetes_pod_container_name} from Kubernetes meta labels.

discovery.kubernetes "all_pods" {
  role = "pod"
  selectors {
    field = "spec.nodeName=" + sys.env("HOSTNAME")
    role = "pod"
  }
}

discovery.relabel "local_pods" {
  targets = discovery.kubernetes.all_pods.targets
  rule {
    action = "drop"
    regex = "Succeeded|Failed"
    source_labels = ["__meta_kubernetes_pod_phase"]
  }
  rule {
    action = "replace"
    regex = "(.*)@(.*)"
    replacement = "ebpf/${1}/${2}"
    separator = "@"
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    target_label = "service_name"
  }
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_namespace"]
    target_label = "namespace"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label = "pod"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label = "node"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label = "container"
  }
}
pyroscope.ebpf "local_pods" {
  forward_to = [ pyroscope.write.endpoint.receiver ]
  targets = discovery.relabel.local_pods.output
}

pyroscope.write "endpoint" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Docker discovery

The following example collects performance profiles from containers discovered by discovery.docker and ignores all other profiles collected from outside any docker container. The service_name label is set to the __meta_docker_container_name label.

discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "local_containers" {
  targets = discovery.docker.linux.targets
  rule {
    action = "replace"
    source_labels = ["__meta_docker_container_name"]
    target_label = "service_name"
  }
}

pyroscope.write "staging" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

pyroscope.ebpf "default" {
  forward_to   = [ pyroscope.write.staging.receiver ]
  targets      = discovery.relabel.local_containers.output
}

Compatible components

pyroscope.ebpf can accept arguments from the following components:

Components that export Targets
Components that export Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

pyroscope.enrich

Mon, 30 Mar 2026 15:47:22 +0000

`pyroscope.enrich`

EXPERIMENTAL: This is an experimental component. Experimental components are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental component, you must set the stability.level flag to experimental.

pyroscope.enrich enriches profiles with additional labels from service discovery targets. It matches a label from incoming profiles against a label from discovered targets, and copies specified labels from the matched target to the profile.

Usage

pyroscope.enrich "<LABEL>" {
  targets = <DISCOVERY_COMPONENT>.targets
  target_match_label = "<LABEL>"
  forward_to = [<RECEIVER_LIST>]
}

Arguments

You can use the following arguments with pyroscope.enrich:

Name	Type	Description	Required
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send enriched profiles to.	yes
`target_match_label`	`string`	The label from discovered targets to match against.	yes
`targets`	`list(Target)`	List of targets from a discovery component.	yes
`labels_to_copy`	`list(string)`	List of labels to copy from discovered targets to profiles. If empty, all labels are copied.	no
`profiles_match_label`	`string`	The label from incoming profiles to match against discovered targets.	no

If profiles_match_label isn’t provided, the component uses target_match_label for matching profile labels.

Blocks

pyroscope.enrich doesn’t support any blocks. Configure this component with arguments.

Exported fields

The following fields are exported and can be referenced by other components:

Name	Type	Description
`receiver`	`ProfilesReceiver`	The receiver for profiles.

Component health

pyroscope.enrich is only reported as unhealthy if given an invalid configuration.

Debug information

pyroscope.enrich doesn’t expose debug information.

Debug metrics

pyroscope.enrich doesn’t expose additional metrics.

Example

This example enriches profiles received over HTTP with metadata from Kubernetes service discovery:

// Discover Kubernetes pods
discovery.kubernetes "pods" {
  role = "pod"
}

// Add custom labels from Kubernetes metadata
discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label  = "node"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_label_environment"]
    target_label  = "environment"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_ip"]
    target_label  = "pod_ip"
  }
}

// Receive profiles over HTTP
pyroscope.receive_http "default" {
  http {
    listen_address = "0.0.0.0"
    listen_port    = 4040
  }
  forward_to = [pyroscope.enrich.metadata.receiver]
}

// Enrich profiles with Kubernetes metadata
pyroscope.enrich "metadata" {
  targets               = discovery.relabel.pods.output
  target_match_label    = "pod_ip"
  profiles_match_label  = "service_name"
  labels_to_copy        = ["namespace", "node", "app", "environment"]
  forward_to            = [pyroscope.write.default.receiver]
}

// Write profiles to Pyroscope
pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Component behavior

The component matches profiles to discovered targets and enriches them with additional labels:

For each profile, it looks up the value of profiles_match_label from the profile’s labels, or target_match_label if profiles_match_label isn’t specified.
It matches this value against the target_match_label in discovered targets.
When it finds a match, it copies the requested labels_to_copy from the discovered target to the profile. If labels_to_copy is empty, it copies all labels.
The component forwards the profile, enriched or unchanged, to the configured receivers.

Caution
By default, pyroscope.enrich is ready as it starts, even if discovery doesn’t find targets. If you send profiles to this component before the metadata synchronizes, the component passes them through as-is, without enrichment. This is most likely to impact pyroscope.enrich on startup for a short time before discovery components send a list of targets.

Compatible components

pyroscope.enrich can accept arguments from the following components:

Components that export Targets
Components that export Pyroscope ProfilesReceiver

pyroscope.enrich has exports that can be consumed by the following components:

Components that consume Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

pyroscope.java

Mon, 13 Apr 2026 07:37:47 +0000

`pyroscope.java`

pyroscope.java continuously profiles Java processes running on the local Linux OS using async-profiler.

Note
To use the pyroscope.java component you must run Alloy as root and inside host PID namespace.

Usage

pyroscope.java "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Target JVM configuration

When you use pyroscope.java to profile Java applications, you can configure the target JVMs with some command line flags that ensure accurate profiling, especially for inlined methods. Add the following flags to your Java application’s startup command:

-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints

For more details, refer to Restrictions/Limitations in the async-profiler documentation.

Additional configuration for Linux capabilities

If your Kubernetes environment has Linux capabilities enabled, configure the following in your Helm values to ensure pyroscope.java functions properly:

alloy:
  securityContext:
    runAsUser: 0
    runAsNonRoot: false
    capabilities:
      add:
        - PERFMON
        - SYS_PTRACE
        - SYS_RESOURCE
        - SYS_ADMIN

These capabilities enable Alloy to access performance monitoring subsystems, trace processes, override resource limits, and perform necessary system administration tasks for profiling.

Note
Adjust capabilities based on your specific security requirements and environment, following the principle of least privilege. The capability behavior depends on Container Runtime Interface (CRI) settings. For example, in Docker, capabilities that aren’t on the allowlist are dropped by default.

Arguments

You can use the following arguments with pyroscope.java:

Name	Type	Description	Default	Required
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send collected profiles to.		yes
`targets`	`list(map(string))`	List of java process targets to profile.		yes
`tmp_dir`	`string`	Temporary directory to store async-profiler.	`"/tmp"`	no

Profiling behavior

The special label __process_pid__ must always be present in each target of targets and corresponds to the PID of the process to profile.

After component startup, pyroscope.java creates a temporary directory under tmp_dir and extracts the async-profiler binaries for both glibc and musl into the directory with the following layout.

/tmp/alloy-asprof-glibc-{SHA1}/bin/asprof
/tmp/alloy-asprof-glibc-{SHA1}/lib/libasyncProfiler.so
/tmp/alloy-asprof-musl-{SHA1}/bin/asprof
/tmp/alloy-asprof-musl-{SHA1}/lib/libasyncProfiler.so

After process profiling startup, the component detects libc type and copies according libAsyncProfiler.so into the target process file system at the exact same path.

Note
The asprof binary runs with root permissions. If you change the tmp_dir configuration to something other than /tmp, then you must ensure that the directory is only writable by root.

The filesystem mounted at tmp_dir in the Alloy and target containers, needs to allow execution of files stored there. Typically a mount option called noexec would prevent files from being executed.

`targets`

The special __process_pid__ label must always be present and corresponds to the process PID that’s used for profiling.

Labels starting with a double underscore (__) are treated as internal, and are removed prior to scraping.

The special label service_name is required and must always be present. If it’s not specified, pyroscope.scrape will attempt to infer it from either of the following sources, in this order:

__meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name Pod annotation.
__meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
__meta_docker_container_name
__meta_dockerswarm_container_label_service_name or __meta_dockerswarm_service_name

If service_name isn’t specified and couldn’t be inferred, then it’s set to unspecified.

Blocks

You can use the following block with pyroscope.java:

Block	Description	Required
profiling_config`	Describes java profiling configuration.	no

`profiling_config`

The profiling_config block describes how async-profiler is invoked.

The following arguments are supported:

Name	Type	Description	Default	Required
`alloc`	`string`	Allocation profiling sampling configuration It’s passed as an `--alloc` argument to async-profiler.	`"512k"`	no
`cpu`	`bool`	A flag to enable CPU profiling, using `itimer` async-profiler event by default.	`true`	no
`event`	`string`	Sets the CPU profiling event. Can be one of `itimer`, `cpu` or `wall`.	`"itimer"`	no
`interval`	`duration`	How frequently to collect profiles from the targets.	`"60s"`	no
`lock`	`string`	Lock profiling sampling configuration. It’s passed as an `--lock` argument to async-profiler.	`"10ms"`	no
`log_level`	`string`	Sets the log level in async profiler. one of `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, or `NONE`.	`"INFO"`	no
`per_thread`	`bool`	Sets per thread mode on async profiler. It’s passed as an `-t` argument to async-profiler.	`false`	no
`quiet`	`bool`	If set, suppresses the `Profiling started/stopped` log message.	`false`	no
`sample_rate`	`int`	CPU profiling sample rate. It’s converted from Hz to interval and passed as an `-i` argument to async-profiler.	`100`	no

Refer to profiler-options for more information about async-profiler configuration.

`event`

The event argument sets the CPU profiling event:

itimer - Default. Uses the setitimer(ITIMER_PROF) syscall, which generates a signal every time a process consumes CPU.
cpu - Uses PMU-case sampling (like Intel PEBS or AMD IBS), can be more accurate than itimer, but it’s not available on every platform.
wall - This samples all threads equally every given period of time regardless of thread status: Running, Sleeping, or Blocked. For example, this can be helpful when profiling application start-up time or IO-intensive processes.

`per_thread`

The per_thread argument sets per thread mode on async profiler. Threads are profiled separately and each stack trace ends with a frame that denotes a single thread.

The Wall-clock profiler (event=wall) is most useful in per-thread mode.

Exported fields

pyroscope.java doesn’t export any fields that can be referenced by other components.

Component health

pyroscope.java is only reported as unhealthy when given an invalid configuration. In those cases, exported fields retain their last healthy values.

Debug information

pyroscope.java doesn’t expose any component-specific debug information.

Debug metrics

pyroscope.java doesn’t expose any component-specific debug metrics.

Examples

Profile every java process on the current host

pyroscope.write "staging" {
  endpoint {
    url = "http://localhost:4040"
  }
}

discovery.process "all" {
  refresh_interval = "60s"
  discover_config {
    cwd = true
    exe = true
    commandline = true
    username = true
    uid = true
    container_id = true
  }
}

discovery.relabel "java" {
  targets = discovery.process.all.targets
  rule {
    action = "keep"
    regex = ".*/java$"
    source_labels = ["__meta_process_exe"]
  }
}

pyroscope.java "java" {
  targets = discovery.relabel.java.output
  forward_to = [pyroscope.write.staging.receiver]
  profiling_config {
    interval = "60s"
    alloc = "512k"
    cpu = true
    sample_rate = 100
    lock = "1ms"
  }
}

Compatible components

pyroscope.java can accept arguments from the following components:

Components that export Targets
Components that export Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

pyroscope.receive_http

Mon, 13 Apr 2026 07:37:47 +0000

`pyroscope.receive_http`

pyroscope.receive_http receives profiles over HTTP and forwards them to pyroscope.* components capable of receiving profiles.

The HTTP API exposed is compatible with both the Pyroscope HTTP ingest API and the pushv1.PusherService Connect API. This allows pyroscope.receive_http to act as a proxy for Pyroscope profiles, enabling flexible routing and distribution of profile data.

Usage

pyroscope.receive_http "<LABEL>" {
  http {
    listen_address = "<LISTEN_ADDRESS>"
    listen_port = "<PORT>"
  }
  forward_to = <RECEIVER_LIST>
}

The component starts an HTTP server supporting the following endpoints:

POST /ingest: Send profiles to the component, which forwards them to the receivers configured in the forward_to argument. The request format must match the format of the Pyroscope ingest API.
POST /push.v1.PusherService/Push: Send profiles to the component, which forwards them to the receivers configured in the forward_to argument. The request format must match the format of the Pyroscope pushv1.PusherService Connect API.

Arguments

You can use the following argument with pyroscope.receive_http:

Name	Type	Description	Default	Required
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send profiles to.		yes

Blocks

You can use the following blocks with pyroscope.receive_http:

Block	Description	Required
profiling_config`	Describes java profiling configuration.	no

`http`

The http block configures the HTTP server.

You can use the following arguments to configure the http block. Any omitted fields take their default values.

Name	Type	Description	Default	Required
`conn_limit`	`int`	Maximum number of simultaneous HTTP connections. Defaults to no limit.	`0`	no
`listen_address`	`string`	Network address on which the server listens for new connections. Defaults to accepting all incoming connections.	`""`	no
`listen_port`	`int`	Port number on which the server listens for new connections.	`8080`	no
`server_idle_timeout`	`duration`	Idle timeout for HTTP server.	`"120s"`	no
`server_read_timeout`	`duration`	Read timeout for HTTP server.	`"30s"`	no
`server_write_timeout`	`duration`	Write timeout for HTTP server.	`"30s"`	no

`tls`

The tls block configures TLS for the HTTP server.

Name	Type	Description	Default	Required
`cert_pem`	`string`	PEM data of the server TLS certificate.	`""`	no
`cert_file`	`string`	Path to the server TLS certificate on disk.	`""`	no
`client_auth_type`	`string`	Client authentication to use.	`"NoClientCert"`	no
`client_ca_file`	`string`	Path to the client CA file on disk to validate requests against.	`""`	no
`client_ca_pem`	`string`	PEM data of the client CA to validate requests against.	`""`	no
`key_file`	`string`	Path to the server TLS key on disk.	`""`	no
`key_pem`	`secret`	PEM data of the server TLS key.	`""`	no

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

cert_pem and cert_file
key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

Exported fields

pyroscope.receive_http doesn’t export any fields.

Component health

pyroscope.receive_http is reported as unhealthy if it’s given an invalid configuration.

Debug metrics

pyroscope_receive_http_tcp_connections (gauge): Current number of accepted TCP connections. pyroscope_receive_http_tcp_connections_limit (gauge): The maximum number of TCP connections that the component can accept. A value of 0 means no limit.

Troubleshoot

Connection limit errors

When using pyroscope.write to push profiles to a pyroscope.receive_http component, you may encounter errors like:

"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"

This typically indicates that the receiving component has reached its TCP connection limit.

To resolve connection limit errors, first diagnose the issue, then apply one of the solutions.

Diagnose connection limit issues

Check the connection metrics on the pyroscope.receive_http component:
- pyroscope_receive_http_tcp_connections: Current number of accepted TCP connections
- pyroscope_receive_http_tcp_connections_limit: Maximum number of TCP connections allowed
If the current connections are approaching or at the limit, you need to take action.

Solution 1: Increase the connection limit

To increase the connection limit, increase the conn_limit parameter in the pyroscope.receive_http configuration:

pyroscope.receive_http "example" {
  http {
    conn_limit = 32768  // Increase from default 16384
    // ... other settings
  }
  // ... rest of configuration
}

Solution 2: Horizontal scaling

To distribute the connection load across multiple receivers, deploy multiple instances of pyroscope.receive_http behind a load balancer.

Timeout chain issues

When chaining multiple Pyroscope components such as pyroscope.write to pyroscope.receive_http to another pyroscope.write, you may encounter timeout issues that prevent retries:

"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"

Understand the problem

This issue occurs when:

The first pyroscope.write component sends profiles to pyroscope.receive_http.
The pyroscope.receive_http component forwards profiles to another pyroscope.write component.
Both pyroscope.write components have the same default remote_timeout of 10 seconds.
The request context passes from the first pyroscope.write through pyroscope.receive_http to the second pyroscope.write, maintaining the original 10-second deadline.
If the second pyroscope.write component’s downstream request takes the full 10 seconds due to a broken TCP idle connection, there’s no time left for retries.

To resolve timeout chain issues, apply one of the following solutions.

Solution 1: Increase timeout on the first pyroscope.write

To provide buffer time for retries, increase the remote_timeout on the initial pyroscope.write component:

pyroscope.write "w1" {
  endpoint {
    url = "http://pyroscope-receiver:8080"
    remote_timeout = "30s"  // Increased from default 10s
    // ... other settings
  }
  // ... rest of configuration
}

Solution 2: Decrease timeout on the downstream pyroscope.write

To ensure faster failures and allow time for retries, reduce the remote_timeout on the downstream pyroscope.write component:

pyroscope.write "w2" {
  endpoint {
    url = "http://pyroscope-backend:4040"
    remote_timeout = "3s"  // Reduced from default 10s
    // ... other settings
  }
  // ... rest of configuration
}

Important considerations

Normal latency: Pyroscope servers with the new architecture typically have 500-1000ms average latency for requests
Timeout buffer: Always leave sufficient buffer time for retries when chaining components
Retry configuration: Consider adjusting max_backoff_retries and backoff periods alongside timeout values
Monitoring: Monitor the pyroscope_write_latency metric to understand actual request latencies and adjust timeouts accordingly

Example

This example creates a pyroscope.receive_http component, which starts an HTTP server listening on 0.0.0.0 and port 9999. The server receives profiles and forwards them to multiple pyroscope.write components, which write these profiles to different HTTP endpoints.

// Receives profiles over HTTP
pyroscope.receive_http "default" {
  http {
    listen_address = "0.0.0.0"
    listen_port = 9999
  }
  forward_to = [pyroscope.write.staging.receiver, pyroscope.write.production.receiver]
}

// Send profiles to a staging Pyroscope instance
pyroscope.write "staging" {
  endpoint {
    url = "http://pyroscope-staging:4040"
  }
}

// Send profiles to a production Pyroscope instance
pyroscope.write "production" {
  endpoint {
    url = "http://pyroscope-production:4040"
  }
}

Note
This example demonstrates forwarding to multiple pyroscope.write components. This configuration duplicates the received profiles and sends a copy to each configured pyroscope.write component.

You can also create multiple pyroscope.receive_http components with different configurations to listen on different addresses or ports as needed. This flexibility allows you to design a setup that best fits your infrastructure and profile routing requirements.

Compatible components

pyroscope.receive_http can accept arguments from the following components:

Components that export Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

pyroscope.relabel

Mon, 13 Apr 2026 07:37:47 +0000

`pyroscope.relabel`

The pyroscope.relabel component rewrites the external label set of each profile passed to its receiver by applying one or more relabeling rules and forwards the results to the list of receivers.

If no rules are defined or applicable to some profiles, then those profiles are forwarded as-is to each receiver passed in the component’s arguments. The profile is dropped if no external labels remain after the relabeling rules are applied.

pyroscope.relabel only rewrites labels that aren’t embedded in the profile itself, such as labels inferred by pyroscope.scrape or labels provided through the /ingest?name=... query parameter. It doesn’t parse or modify labels embedded inside profile payloads like pprof sample labels.

The most common use of pyroscope.relabel is to filter profiles or standardize external labels passed to one or more downstream receivers. The rule blocks are applied to the label set of each profile in order of their appearance in the configuration file.

Usage

pyroscope.relabel "<LABEL>" {
    forward_to = <RECEIVER_LIST>

    rule {
        ...
    }

    ...
}

Arguments

You can use the following arguments with pyroscope.relabel:

Name	Type	Description	Default	Required
`forward_to`	`list(pyroscope.Appendable)`	List of receivers to forward profiles to after relabeling		yes
`max_cache_size`	`number`	Maximum number of entries in the label cache	`10000`	no

Blocks

You can use the following block with pyroscope.relabel:

Block	Description	Required
profiling_config`	Describes java profiling configuration.	no

`rule`

The rule block contains the definition of any relabeling rules that can be applied to an input metric. If more than one rule block is defined, the transformations are applied in top-down order.

The following arguments can be used to configure a rule. All arguments are optional. Omitted fields take their default values.

Name	Type	Description	Default	Required
`action`	`string`	The relabeling action to perform.	replace	no
`modulus`	`uint`	A positive integer used to calculate the modulus of the hashed source label values.		no
`regex`	`string`	A valid RE2 expression with support for parenthesized capture groups. Used to match the extracted value from the combination of the `source_label` and `separator` fields or filter labels during the `labelkeep/labeldrop/labelmap` actions.	`(.*)`	no
`replacement`	`string`	The value against which a regular expression replace is performed, if the regular expression matches the extracted value. Supports previously captured groups.	`"$1"`	no
`separator`	`string`	The separator used to concatenate the values present in `source_labels`.	;	no
`source_labels`	`list(string)`	The list of labels whose values are to be selected. Their content is concatenated using the `separator` and matched against `regex`.		no
`target_label`	`string`	Label to which the resulting value will be written to.		no

You can use the following actions:

drop - Drops metrics where regex matches the string extracted using the source_labels and separator.
dropequal - Drop targets for which the concatenated source_labels do match target_label.
hashmod - Hashes the concatenated labels, calculates its modulo modulus and writes the result to the target_label.
keep - Keeps metrics where regex matches the string extracted using the source_labels and separator.
keepequal - Drop targets for which the concatenated source_labels don’t match target_label.
labeldrop - Matches regex against all label names. Any labels that match are removed from the metric’s label set.
labelkeep - Matches regex against all label names. Any labels that don’t match are removed from the metric’s label set.
labelmap - Matches regex against all label names. Any labels that match are renamed according to the contents of the replacement field.
lowercase - Sets target_label to the lowercase form of the concatenated source_labels.
replace - Matches regex to the concatenated labels. If there’s a match, it replaces the content of the target_label using the contents of the replacement field.
uppercase - Sets target_label to the uppercase form of the concatenated source_labels.

Note
The regular expression capture groups can be referred to using either the $CAPTURE_GROUP_NUMBER or ${CAPTURE_GROUP_NUMBER} notation.

Exported fields

The following fields are exported and can be referenced by other components:

Name	Type	Description
`receiver`	`ProfilesReceiver`	A receiver that accepts profiles for relabeling.
`rules`	`[]relabel.Config`	The list of relabeling rules.

Component health

pyroscope.relabel is reported as unhealthy if it is given an invalid configuration.

Debug metrics

pyroscope_relabel_cache_hits (counter): Total number of cache hits.
pyroscope_relabel_cache_misses (counter): Total number of cache misses.
pyroscope_relabel_cache_size (gauge): Total size of relabel cache.
pyroscope_relabel_profiles_dropped (counter): Total number of profiles dropped by relabeling rules.
pyroscope_relabel_profiles_processed (counter): Total number of profiles processed.
pyroscope_relabel_profiles_written (counter): Total number of profiles forwarded.

Example

pyroscope.receive_http "default" {
    forward_to = [pyroscope.relabel.filter_profiles.receiver]

    http {
        listen_address = "0.0.0.0"
        listen_port = 9999
    }
}

pyroscope.relabel "filter_profiles" {
    forward_to = [pyroscope.write.staging.receiver]

    // This creates a consistent hash value (0 or 1) for each unique combination of labels
    // Using multiple source labels provides better sampling distribution across your profiles
    rule {
        source_labels = ["env"]
        target_label = "__tmp_hash"
        action = "hashmod"
        modulus = 2
    }

    // This effectively samples ~50% of profile series
    // The same combination of source label values will always hash to the same number,
    // ensuring consistent sampling
    rule {
        source_labels = ["__tmp_hash"]
        action       = "drop"
        regex        = "^1$"
    }
}

pyroscope.write "staging" {
  endpoint {
    url = "http://pyroscope-staging:4040"
  }
}

Compatible components

pyroscope.relabel can accept arguments from the following components:

Components that export Pyroscope ProfilesReceiver

pyroscope.relabel has exports that can be consumed by the following components:

Components that consume Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

pyroscope.scrape

Mon, 13 Apr 2026 07:37:47 +0000

`pyroscope.scrape`

pyroscope.scrape collects [pprof] performance profiles for a given set of HTTP targets.

pyroscope.scrape mimics the scraping behavior of prometheus.scrape. Similarly to how Prometheus scrapes metrics via HTTP, pyroscope.scrape collects profiles via HTTP requests.

Unlike Prometheus, which usually only scrapes one /metrics endpoint per target, pyroscope.scrape may need to scrape multiple endpoints for the same target. This is because different types of profiles are scraped on different endpoints. For example, mutex profiles may be scraped on a /debug/pprof/delta_mutex HTTP endpoint, whereas memory consumption may be scraped on a /debug/pprof/allocs HTTP endpoint.

The profile paths, protocol scheme, scrape interval, scrape timeout, query parameters, as well as any other settings can be configured within pyroscope.scrape.

The pyroscope.scrape component regards a scrape as successful if it responded with an HTTP 200 OK status code and returned the body of a valid [pprof] profile.

If a scrape request fails, the debug UI for pyroscope.scrape will show:

Detailed information about the failure.
The time of the last successful scrape.
The labels last used for scraping.

The scraped performance profiles can be forwarded to components such as pyroscope.write via the forward_to argument.

Multiple pyroscope.scrape components can be specified by giving them different labels.

Usage

pyroscope.scrape "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Arguments

pyroscope.scrape starts a new scrape job to scrape all of the input targets. Multiple scrape jobs can be started for a single input target when scraping multiple profile types.

You can use the following arguments with pyroscope.scrape:

Name	Type	Description	Default	Required
`targets`	`list(map(string))`	List of targets to scrape.		yes
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send scraped profiles to.		yes
`job_name`	`string`	The job name to override the job label with.	component name	no
`params`	`map(list(string))`	A set of query parameters with which the target is scraped.		no
`scrape_interval`	`duration`	How frequently to scrape the targets of this scrape configuration.	`"15s"`	no
`scrape_timeout`	`duration`	The timeout for scraping targets of this configuration. Must be larger than `scrape_interval`.	`"18s"`	no
`delta_profiling_duration`	`duration`	The duration for a delta profiling to be scraped. Must be larger than 1 second.	`"14s"`	no
`scheme`	`string`	The URL scheme with which to fetch metrics from targets.	`"http"`	no
`bearer_token_file`	`string`	File containing a bearer token to authenticate with.		no
`bearer_token`	`secret`	Bearer token to authenticate with.		no
`enable_http2`	`bool`	Whether HTTP2 is supported for requests.	`true`	no
`follow_redirects`	`bool`	Whether redirects returned by the server should be followed.	`true`	no
`http_headers`	`map(list(secret))`	Custom HTTP headers to be sent along with each request. The map key is the header name.		no
`proxy_url`	`string`	HTTP proxy to send requests through.		no
`no_proxy`	`string`	Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.		no
`proxy_from_environment`	`bool`	Use the proxy URL indicated by environment variables.	`false`	no
`proxy_connect_header`	`map(list(secret))`	Specifies headers to send to proxies during CONNECT requests.		no

At most, one of the following can be provided:

[authorization][authorization] block
[basic_auth][basic_auth] block
bearer_token_file argument
bearer_token argument
[oauth2][oauth2] block

Any omitted arguments take on their default values. If conflicting arguments are being passed, for example, configuring both bearer_token and bearer_token_file, then pyroscope.scrape will fail to start and will report an error.

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_from_environment uses the environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). Requests use the proxy from the environment variable matching their scheme, unless excluded by NO_PROXY. proxy_url and no_proxy must not be configured if proxy_from_environment is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

`job_name`

The job_name argument defaults to the component’s unique identifier.

For example, the job_name of pyroscope.scrape "local" { ... } will be "pyroscope.scrape.local".

`targets`

The list of targets can be provided statically, dynamically, or a combination of both.

The following special labels can change the behavior of pyroscope.scrape:

__address__ is the special label that must always be present and corresponds to the <host>:<port> that is used for the scrape request.
__name__ is the special label that indicates the profile type being collected.
__profile_path__ is the special label that holds the path to the profile endpoint on the target (e.g. “/debug/pprof/allocs”).
__profile_path_prefix__ is the special label that holds an optional prefix to prepend to the profile path (e.g. “/mimir-prometheus”).
service_name is a required label that identifies the service being profiled.

Labels starting with a double underscore (__) are treated as internal, and are removed prior to scraping.

The special label service_name is required and must always be present. If it’s not specified, pyroscope.scrape will attempt to infer it from either of the following sources, in this order:

__meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name pod annotation.
__meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
__meta_docker_container_name
__meta_dockerswarm_container_label_service_name or __meta_dockerswarm_service_name

If service_name isn’t specified and couldn’t be inferred, then it’s set to unspecified.

The following labels are automatically injected to the scraped profiles so that they can be linked to a scrape target:

Label	Description
`"job"`	The `job_name` that the target belongs to.
`"instance"`	The `__address__` or `<host>:<port>` of the scrape target’s URL.
`"service_name"`	The inferred Pyroscope service name.

`scrape_interval`

The scrape_interval typically refers to the frequency with which Alloy collects performance profiles from the monitored targets. It represents the time interval between consecutive scrapes or data collection events. This parameter is important for controlling the trade-off between resource usage and the freshness of the collected data.

If scrape_interval is short:

Advantages:
- Fewer profiles may be lost if the application being scraped crashes.
Disadvantages:
- Greater consumption of CPU, memory, and network resources during scrapes and remote writes.
- The backend database (Pyroscope) consumes more storage space.

If scrape_interval is long:

Advantages:
- Lower resource consumption.
Disadvantages:
- More profiles may be lost if the application being scraped crashes.
- If the [delta argument][] is set to true, the batch size of each remote write to Pyroscope may be bigger. The Pyroscope database may need to be tuned with higher limits.
- If the [delta argument][] is set to true, there is a larger risk of reaching the HTTP server timeouts of the application being scraped.

For example, consider this situation:

pyroscope.scrape is configured with a scrape_interval of "60s".
The application being scraped is running an HTTP server with a timeout of 30 seconds.
Any scrape HTTP requests where the [delta argument][] is set to true will fail, because they will attempt to run for 59 seconds.

Blocks

You can use the following blocks with pyroscope.scrape:

Block	Description	Required
profiling_config`	Describes java profiling configuration.	no

Any omitted blocks take on their default values. For example, if profile.mutex isn’t specified in the configuration, the defaults documented in [profile.mutex][] are used.

`authorization`

Name	Type	Description	Required
`credentials_file`	`string`	File containing the secret value.	no
`credentials`	`secret`	Secret value.	no
`type`	`string`	Authorization type, for example, “Bearer”.	no

credential and credentials_file are mutually exclusive, and only one can be provided inside an authorization block.

Warning
Using credentials_file causes the file to be read on every outgoing request. Use the local.file component with the credentials attribute instead to avoid unnecessary reads.

`basic_auth`

Name	Type	Description	Required
`password_file`	`string`	File containing the basic auth password.	no
`password`	`secret`	Basic auth password.	no
`username`	`string`	Basic auth username.	no

password and password_file are mutually exclusive, and only one can be provided inside a basic_auth block.

Warning
Using password_file causes the file to be read on every outgoing request. Use the local.file component with the password attribute instead to avoid unnecessary reads.

`clustering`

Name	Type	Description	Default	Required
`enabled`	`bool`	Enables sharing targets with other cluster nodes.	`false`	yes

When Alloy is using clustering, and enabled is set to true, then this pyroscope.scrape component instance opts-in to participating in the cluster to distribute scrape load between all cluster nodes.

Clustering causes the set of targets to be locally filtered down to a unique subset per node, where each node is roughly assigned the same number of targets. If the state of the cluster changes, such as a new node joins, then the subset of targets to scrape per node is recalculated.

When clustering mode is enabled, all Alloy instances participating in the cluster must use the same configuration file and have access to the same service discovery APIs.

If Alloy is not running in clustered mode, this block is a no-op.

`oauth2`

Name	Type	Description	Default	Required
`client_id`	`string`	OAuth2 client ID.		no
`client_secret_file`	`string`	File containing the OAuth2 client secret.		no
`client_secret`	`secret`	OAuth2 client secret.		no
`endpoint_params`	`map(string)`	Optional parameters to append to the token URL.		no
`no_proxy`	`string`	Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.		no
`proxy_connect_header`	`map(list(secret))`	Specifies headers to send to proxies during CONNECT requests.		no
`proxy_from_environment`	`bool`	Use the proxy URL indicated by environment variables.	`false`	no
`proxy_url`	`string`	HTTP proxy to send requests through.		no
`scopes`	`list(string)`	List of scopes to authenticate with.		no
`token_url`	`string`	URL to fetch the token from.		no

client_secret and client_secret_file are mutually exclusive, and only one can be provided inside an oauth2 block.

Warning
Using client_secret_file causes the file to be read on every outgoing request. Use the local.file component with the client_secret attribute instead to avoid unnecessary reads.

The oauth2 block may also contain a separate tls_config sub-block.

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

`profiling_config`

The profiling_config block configures the profiling settings when scraping targets.

The following arguments are supported:

Name	Type	Description	Default	Required
`path_prefix`	`string`	The path prefix to use when scraping targets.		no

`profile.block`

The profile.block block collects profiles on process blocking.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`false`	no
`enabled`	`boolean`	Enable this profile type to be scraped.	`true`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/block"`	no

For more information about the delta argument, see the [delta argument][] section.

`profile.custom`

The profile.custom block allows for collecting profiles from custom endpoints. Blocks must be specified with a label:

profile.custom "<PROFILE_TYPE>" {
  enabled = true
  path    = "<PROFILE_PATH>"
}

You can specify multiple profile.custom blocks. Labels assigned to profile.custom blocks must be unique across the component.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`false`	no
`enabled`	`boolean`	Enable this profile type to be scraped.		yes
`path`	`string`	The path to the profile type on the target.		yes

When the delta argument is true, a seconds query parameter is automatically added to requests. The seconds used will be equal to scrape_interval - 1.

`profile.fgprof`

The profile.fgprof block collects profiles from an [fgprof][] endpoint.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`true`	no
`enabled`	`boolean`	Enable this profile type to be scraped.	`false`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/fgprof"`	no

For more information about the delta argument, see the [delta argument][] section.

`profile.godeltaprof_block`

The profile.godeltaprof_block block collects profiles from [godeltaprof][] block endpoint. The delta is computed on the target.

The following arguments are supported:

Name	Type	Description	Default	Required
`enabled`	`boolean`	Enable this profile type to be scraped.	`false`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/delta_block"`	no

`profile.godeltaprof_memory`

The profile.godeltaprof_memory block collects profiles from [godeltaprof][] memory endpoint. The delta is computed on the target.

The following arguments are supported:

Name	Type	Description	Default	Required
`enabled`	`boolean`	Enable this profile type to be scraped.	`false`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/delta_heap"`	no

`profile.godeltaprof_mutex`

The profile.godeltaprof_mutex block collects profiles from [godeltaprof][] mutex endpoint. The delta is computed on the target.

The following arguments are supported:

Name	Type	Description	Default	Required
`enabled`	`boolean`	Enable this profile type to be scraped.	`false`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/delta_mutex"`	no

`profile.goroutine`

The profile.goroutine block collects profiles on the number of goroutines.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`false`	no
`enabled`	`boolean`	Enable this profile type to be scraped.	`true`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/goroutine"`	no

Refer to [delta argument][] for more information about the delta argument.

`profile.memory`

The profile.memory block collects profiles on memory consumption.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`false`	no
`enabled`	`boolean`	Enable this profile type to be scraped.	`true`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/allocs"`	no

Refer to [delta argument][] for more information about the delta argument.

`profile.mutex`

The profile.mutex block collects profiles on mutexes.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`false`	no
`enabled`	`boolean`	Enable this profile type to be scraped.	`true`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/mutex"`	no

Refer to [delta argument][] for more information about the delta argument.

`profile.process_cpu`

The profile.process_cpu block collects profiles on CPU consumption for the process.

The following arguments are supported:

Name	Type	Description	Default	Required
`delta`	`boolean`	Whether to scrape the profile as a delta.	`true`	no
`enabled`	`boolean`	Enable this profile type to be scraped.	`true`	no
`path`	`string`	The path to the profile type on the target.	`"/debug/pprof/profile"`	no

For more information about the delta argument, see the [delta argument][] section.

`tls_config`

Name	Type	Description	Required
`ca_pem`	`string`	CA PEM-encoded text to validate the server with.	no
`ca_file`	`string`	CA certificate to validate the server with.	no
`cert_pem`	`string`	Certificate PEM-encoded text for client authentication.	no
`cert_file`	`string`	Certificate file for client authentication.	no
`insecure_skip_verify`	`bool`	Disables validation of the server certificate.	no
`key_file`	`string`	Key file for client authentication.	no
`key_pem`	`secret`	Key PEM-encoded text for client authentication.	no
`min_version`	`string`	Minimum acceptable TLS version.	no
`server_name`	`string`	ServerName extension to indicate the name of the server.	no

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

ca_pem and ca_file
cert_pem and cert_file
key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

When min_version isn’t provided, the minimum acceptable TLS version is inherited from Go’s default minimum version, TLS 1.2. If min_version is provided, it must be set to one of the following strings:

"TLS10" (TLS 1.0)
"TLS11" (TLS 1.1)
"TLS12" (TLS 1.2)
"TLS13" (TLS 1.3)

Common configuration

`delta` argument

When the delta argument is false, the [pprof][] HTTP query will be instantaneous.

When the delta argument is true:

The [pprof][] HTTP query runs for a certain amount of time.
A seconds parameter is automatically added to the HTTP request.
The default value for the seconds query parameter is scrape_interval - 1. If you set delta_profiling_duration, then seconds is assigned the same value as delta_profiling_duration. However, the delta_profiling_duration can’t be larger than scrape_interval. For example, if you set scrape_interval to "15s", then seconds defaults to 14s If you set delta_profiling_duration to 16s, then scrape_interval must be set to at least 17s. If the HTTP endpoint is /debug/pprof/profile, then the HTTP query becomes /debug/pprof/profile?seconds=14

Exported fields

pyroscope.scrape doesn’t export any fields that can be referenced by other components.

Component health

pyroscope.scrape is only reported as unhealthy if given an invalid configuration.

Debug information

pyroscope.scrape reports the status of the last scrape for each configured scrape job on the component’s debug endpoint.

Debug metrics

pyroscope_fanout_latency (histogram): Write latency for sending to direct and indirect components.

Examples

Default endpoints of static targets

The following example sets up a scrape job of a statically configured list of targets - Alloy itself and Pyroscope. The scraped profiles are sent to pyroscope.write which remote writes them to a Pyroscope database.

pyroscope.scrape "local" {
  targets = [
    {"__address__" = "localhost:4040", "service_name"="pyroscope"},
    {"__address__" = "localhost:12345", "service_name"="alloy"},
  ]

  forward_to = [pyroscope.write.local.receiver]
}

pyroscope.write "local" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

These endpoints will be scraped every 15 seconds:

http://localhost:4040/debug/pprof/allocs
http://localhost:4040/debug/pprof/block
http://localhost:4040/debug/pprof/goroutine
http://localhost:4040/debug/pprof/mutex
http://localhost:4040/debug/pprof/profile?seconds=14

http://localhost:12345/debug/pprof/allocs
http://localhost:12345/debug/pprof/block
http://localhost:12345/debug/pprof/goroutine
http://localhost:12345/debug/pprof/mutex
http://localhost:12345/debug/pprof/profile?seconds=14

seconds=14 is added to the /debug/pprof/profile endpoint, because:

The delta argument of the profile.process_cpu block is true by default.
scrape_interval is "15s" by default.

The /debug/fgprof endpoint won’t be scraped, because the enabled argument of the profile.fgprof block is false by default.

Default endpoints of dynamic targets

discovery.http "dynamic_targets" {
  url = "https://example.com/scrape_targets"
  refresh_interval = "15s"
}

pyroscope.scrape "local" {
  targets = [discovery.http.dynamic_targets.targets]

  forward_to = [pyroscope.write.local.receiver]
}

pyroscope.write "local" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Default endpoints of static and dynamic targets

discovery.http "dynamic_targets" {
  url = "https://example.com/scrape_targets"
  refresh_interval = "15s"
}

pyroscope.scrape "local" {
  targets = array.concat([
    {"__address__" = "localhost:4040", "service_name"="pyroscope"},
    {"__address__" = "localhost:12345", "service_name"="alloy"},
  ], discovery.http.dynamic_targets.targets)

  forward_to = [pyroscope.write.local.receiver]
}

pyroscope.write "local" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Enable and disable profiles

pyroscope.scrape "local" {
  targets = [
    {"__address__" = "localhost:12345", "service_name"="alloy"},
  ]

  profiling_config {
    profile.fgprof {
      enabled = true
    }
    profile.block {
      enabled = false
    }
    profile.mutex {
      enabled = false
    }
  }

  forward_to = [pyroscope.write.local.receiver]
}

These endpoints will be scraped every 15 seconds:

http://localhost:12345/debug/pprof/allocs
http://localhost:12345/debug/pprof/goroutine
http://localhost:12345/debug/pprof/profile?seconds=14
http://localhost:12345/debug/fgprof?seconds=14

These endpoints will NOT be scraped because they are explicitly disabled:

http://localhost:12345/debug/pprof/block
http://localhost:12345/debug/pprof/mutex

Compatible components

pyroscope.scrape can accept arguments from the following components:

Components that export Targets
Components that export Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

pyroscope.write

Mon, 13 Apr 2026 07:37:47 +0000

`pyroscope.write`

pyroscope.write receives performance profiles from other components and forwards them to a series of user-supplied endpoints. When pyroscope.write forwards profiles, all labels starting with double underscore (__) are dropped before the data is sent, with the following exceptions:

__name__ is preserved because it identifies the profile type.
__delta__is preserved because it’s required for delta profiles.

You can specify multiple pyroscope.write components by giving them different labels.

Usage

pyroscope.write "<LABEL>" {
  endpoint {
    url = "<PYROSCOPE_URL>"

    ...
  }

  ...
}

Arguments

You can use the following argument with pyroscope.write:

Name	Type	Description	Default	Required
`external_labels`	`map(string)`	Labels to add to profiles sent over the network.		no

Blocks

You can use the following blocks with pyroscope.write:

Block	Description	Required
profiling_config`	Describes java profiling configuration.	no

`endpoint`

The endpoint block describes a single location to send profiles to. Multiple endpoint blocks can be provided to send profiles to multiple locations.

The following arguments are supported:

Name	Type	Description	Default	Required
`url`	`string`	Full URL to send metrics to.		yes
`bearer_token_file`	`string`	File containing a bearer token to authenticate with.		no
`bearer_token`	`secret`	Bearer token to authenticate with.		no
`enable_http2`	`bool`	Whether HTTP2 is supported for requests.	`true`	no
`follow_redirects`	`bool`	Whether redirects returned by the server should be followed.	`true`	no
`headers`	`map(string)`	Extra headers to deliver with the request.		no
`http_headers`	`map(list(secret))`	Custom HTTP headers to be sent along with each request. The map key is the header name.		no
`max_backoff_period`	`duration`	Maximum backoff time between retries.	`"5m"`	no
`max_backoff_retries`	`int`	Maximum number of retries. 0 to retry infinitely.	`10`	no
`min_backoff_period`	`duration`	Initial backoff time between retries.	`"500ms"`	no
`name`	`string`	Optional name to identify the endpoint in metrics.		no
`no_proxy`	`string`	Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.		no
`proxy_connect_header`	`map(list(secret))`	Specifies headers to send to proxies during CONNECT requests.		no
`proxy_from_environment`	`bool`	Use the proxy URL indicated by environment variables.	`false`	no
`proxy_url`	`string`	HTTP proxy to send requests through.		no
`remote_timeout`	`duration`	Timeout for requests made to the URL.	`"10s"`	no

At most, one of the following can be provided:

[authorization][authorization] block
[basic_auth][basic_auth] block
[bearer_token_file][endpoint] argument
[bearer_token][endpoint] argument
[oauth2][oauth2] block

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

When you provide multiple endpoint blocks, profiles are concurrently forwarded to all configured locations.

`authorization`

Name	Type	Description	Required
`credentials_file`	`string`	File containing the secret value.	no
`credentials`	`secret`	Secret value.	no
`type`	`string`	Authorization type, for example, “Bearer”.	no

credential and credentials_file are mutually exclusive, and only one can be provided inside an authorization block.

Warning
Using credentials_file causes the file to be read on every outgoing request. Use the local.file component with the credentials attribute instead to avoid unnecessary reads.

`basic_auth`

Name	Type	Description	Required
`password_file`	`string`	File containing the basic auth password.	no
`password`	`secret`	Basic auth password.	no
`username`	`string`	Basic auth username.	no

password and password_file are mutually exclusive, and only one can be provided inside a basic_auth block.

Warning
Using password_file causes the file to be read on every outgoing request. Use the local.file component with the password attribute instead to avoid unnecessary reads.

`oauth2`

Name	Type	Description	Default	Required
`client_id`	`string`	OAuth2 client ID.		no
`client_secret_file`	`string`	File containing the OAuth2 client secret.		no
`client_secret`	`secret`	OAuth2 client secret.		no
`endpoint_params`	`map(string)`	Optional parameters to append to the token URL.		no
`no_proxy`	`string`	Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying.		no
`proxy_connect_header`	`map(list(secret))`	Specifies headers to send to proxies during CONNECT requests.		no
`proxy_from_environment`	`bool`	Use the proxy URL indicated by environment variables.	`false`	no
`proxy_url`	`string`	HTTP proxy to send requests through.		no
`scopes`	`list(string)`	List of scopes to authenticate with.		no
`token_url`	`string`	URL to fetch the token from.		no

client_secret and client_secret_file are mutually exclusive, and only one can be provided inside an oauth2 block.

Warning
Using client_secret_file causes the file to be read on every outgoing request. Use the local.file component with the client_secret attribute instead to avoid unnecessary reads.

The oauth2 block may also contain a separate tls_config sub-block.

no_proxy can contain IPs, CIDR notations, and domain names. IP and domain names can contain port numbers. proxy_url must be configured if no_proxy is configured.

proxy_connect_header should only be configured if proxy_url or proxy_from_environment are configured.

`tls_config`

Name	Type	Description	Required
`ca_pem`	`string`	CA PEM-encoded text to validate the server with.	no
`ca_file`	`string`	CA certificate to validate the server with.	no
`cert_pem`	`string`	Certificate PEM-encoded text for client authentication.	no
`cert_file`	`string`	Certificate file for client authentication.	no
`insecure_skip_verify`	`bool`	Disables validation of the server certificate.	no
`key_file`	`string`	Key file for client authentication.	no
`key_pem`	`secret`	Key PEM-encoded text for client authentication.	no
`min_version`	`string`	Minimum acceptable TLS version.	no
`server_name`	`string`	ServerName extension to indicate the name of the server.	no

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

ca_pem and ca_file
cert_pem and cert_file
key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

"TLS10" (TLS 1.0)
"TLS11" (TLS 1.1)
"TLS12" (TLS 1.2)
"TLS13" (TLS 1.3)

Exported fields

The following fields are exported and can be referenced by other components:

Name	Type	Description
`receiver`	`receiver`	A value that other components can use to send profiles to.

Component health

pyroscope.write is only reported as unhealthy if given an invalid configuration. In those cases, exported fields are kept at their last healthy values.

Debug information

pyroscope.write doesn’t expose any component-specific debug information.

Metrics

pyroscope.write exposes the following metrics:

Metric	Type	Description
`pyroscope_write_sent_bytes_total`	Counter	Total number of compressed bytes sent to Pyroscope endpoints.
`pyroscope_write_dropped_bytes_total`	Counter	Total number of compressed bytes dropped by Pyroscope endpoints.
`pyroscope_write_sent_profiles_total`	Counter	Total number of profiles sent to Pyroscope endpoints.
`pyroscope_write_dropped_profiles_total`	Counter	Total number of profiles dropped by Pyroscope endpoints.
`pyroscope_write_retries_total`	Counter	Total number of retries to Pyroscope endpoints.
`pyroscope_write_latency`	Histogram	Write latency for sending profiles to Pyroscope endpoints.

All metrics include an endpoint label identifying the specific endpoint URL. The pyroscope_write_latency metric includes an additional type label with the following values:

push_total: Total latency for push operations
push_endpoint: Per-endpoint latency for push operations
push_downstream: Downstream request latency for push operations
ingest_total: Total latency for ingest operations
ingest_endpoint: Per-endpoint latency for ingest operations
ingest_downstream: Downstream request latency for ingest operations

Troubleshoot

Connection limit errors

When using pyroscope.write to push profiles to a pyroscope.receive_http component, you may encounter errors like:

"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"

This typically indicates that the receiving component has reached its TCP connection limit.

To resolve connection limit errors, first diagnose the issue, then apply one of the solutions.

Diagnose connection limit issues

Check the connection metrics on the pyroscope.receive_http component:
- pyroscope_receive_http_tcp_connections: Current number of accepted TCP connections
- pyroscope_receive_http_tcp_connections_limit: Maximum number of TCP connections allowed
If the current connections are approaching or at the limit, you need to take action.

Solution 1: Increase the connection limit

To increase the connection limit, increase the conn_limit parameter in the pyroscope.receive_http configuration:

pyroscope.receive_http "example" {
  http {
    conn_limit = 32768  // Increase from default 16384
    // ... other settings
  }
  // ... rest of configuration
}

Solution 2: Horizontal scaling

To distribute the connection load across multiple receivers, deploy multiple instances of pyroscope.receive_http behind a load balancer.

Timeout chain issues

When chaining multiple Pyroscope components such as pyroscope.write to pyroscope.receive_http to another pyroscope.write, you may encounter timeout issues that prevent retries:

"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"

Understand the problem

This issue occurs when:

The first pyroscope.write component sends profiles to pyroscope.receive_http.
The pyroscope.receive_http component forwards profiles to another pyroscope.write component.
Both pyroscope.write components have the same default remote_timeout of 10 seconds.
The request context passes from the first pyroscope.write through pyroscope.receive_http to the second pyroscope.write, maintaining the original 10-second deadline.
If the second pyroscope.write component’s downstream request takes the full 10 seconds due to a broken TCP idle connection, there’s no time left for retries.

To resolve timeout chain issues, apply one of the following solutions.

Solution 1: Increase timeout on the first pyroscope.write

To provide buffer time for retries, increase the remote_timeout on the initial pyroscope.write component:

pyroscope.write "w1" {
  endpoint {
    url = "http://pyroscope-receiver:8080"
    remote_timeout = "30s"  // Increased from default 10s
    // ... other settings
  }
  // ... rest of configuration
}

Solution 2: Decrease timeout on the downstream pyroscope.write

To ensure faster failures and allow time for retries, reduce the remote_timeout on the downstream pyroscope.write component:

pyroscope.write "w2" {
  endpoint {
    url = "http://pyroscope-backend:4040"
    remote_timeout = "3s"  // Reduced from default 10s
    // ... other settings
  }
  // ... rest of configuration
}

Important considerations

Normal latency: Pyroscope servers with the new architecture typically have 500-1000ms average latency for requests
Timeout buffer: Always leave sufficient buffer time for retries when chaining components
Retry configuration: Consider adjusting max_backoff_retries and backoff periods alongside timeout values
Monitoring: Monitor the pyroscope_write_latency metric to understand actual request latencies and adjust timeouts accordingly

Example

pyroscope.write "staging" {
  // Send metrics to a locally running Pyroscope instance.
  endpoint {
    url = "http://pyroscope:4040"
    headers = {
      "X-Scope-OrgID" = "squad-1",
    }
  }
  external_labels = {
    "env" = "staging",
  }
}

pyroscope.scrape "default" {
  targets = [
    {"__address__" = "pyroscope:4040", "service_name"="pyroscope"},
    {"__address__" = "alloy:12345", "service_name"="alloy"},
  ]
  forward_to = [pyroscope.write.staging.receiver]
}

Compatible components

pyroscope.write has exports that can be consumed by the following components:

Components that consume Pyroscope ProfilesReceiver

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.