Grafana Alloy on Grafana Labs

Receive profiles from Pyroscope SDKs

Wed, 08 Apr 2026 14:38:28 +0000

Receive profiles from Pyroscope SDKs

The pyroscope.receive_http component in Alloy receives profiles from applications instrumented with Pyroscope SDKs. This approach provides several benefits:

Lower latency by sending profiles to a local Alloy instance instead of over internet
Separation of infrastructure concerns (auth, routing) from application code
Centralized management of authentication and metadata enrichment

For more information about this component, refer to the pyroscope.receive_http component documentation.

Note
The pyroscope.receive_http component is currently in public preview. To use this component, set the --stability.level flag to public-preview. For more information about Alloy’s run usage, refer to the run command documentation documentation.

To set up profile receiving, you need to:

Configure Alloy components
Configure your application’s SDK
Start Alloy

Configure Alloy components

The configuration requires at least two components:

pyroscope.receive_http to receive profiles via HTTP
pyroscope.write to forward profiles to Pyroscope

Here’s a basic configuration that sets up a simple profile collection pipeline. It creates a receiver to collect profiles from your applications and forwards them through a writer component to send them to the Pyroscope backend:

// Receives profiles over HTTP
pyroscope.receive_http "default" {
   http {
       listen_address = "0.0.0.0"
       listen_port = 9999
   }
   forward_to = [pyroscope.write.backend.receiver]
}

// Forwards profiles to Pyroscope
pyroscope.write "backend" {
   endpoint {
       url = "http://pyroscope:4040"
   }
}

Configure application SDK

Update your application’s SDK configuration to point to Alloy’s receive endpoint instead of Pyroscope directly. For example, in Go:

config := pyroscope.Config{
    ApplicationName: "my.service.cpu",
    ServerAddress:   "http://localhost:9999", // Alloy's receive endpoint
}

Check your specific language SDK documentation for the exact configuration options.

Examples

The examples in this section provide samples you can use as a starting point for your own configurations.

Explore the example in the Pyroscope GitHub repository to learn how to configure Grafana Alloy to receive profiles from a Golang application instrumented with Pyroscope.

Basic receiving setup

This example shows a basic setup receiving profiles on port 9090 and forwarding them to a local Pyroscope instance:

pyroscope.receive_http "default" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9090
    }
    forward_to = [pyroscope.write.production.receiver]
}

pyroscope.write "production" {
    endpoint {
        url = "http://localhost:4040"
    }
}

Authentication

To send profiles to an authenticated Pyroscope endpoint:

pyroscope.write "production" {
    endpoint {
        url = "http://pyroscope:4040"
        basic_auth {
            username = env("PYROSCOPE_USERNAME")
            password = env("PYROSCOPE_PASSWORD")
        }
    }
}

Adding external labels

External labels are added to all profiles forwarded through the write component. This is useful for adding infrastructure metadata:

pyroscope.receive_http "default" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9999
    }
    forward_to = [pyroscope.write.backend.receiver]
}

pyroscope.write "backend" {
    endpoint {
        url = "http://pyroscope:4040"
    }
    external_labels = {
        "env"      = "production",
        "region"   = "us-west-1",
        "instance" = env("HOSTNAME"),
        "cluster"  = "main",
    }
}

Multiple destinations

Forward received profiles to multiple destinations - useful for testing or migration scenarios:

pyroscope.receive_http "default" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9999
    }
    forward_to = [pyroscope.write.staging.receiver, pyroscope.write.production.receiver]
}

// Send profiles to staging
pyroscope.write "staging" {
    endpoint {
        url = "http://pyroscope-staging:4040"
    }
    external_labels = {
        "env" = "staging",
    }
}

// Send profiles to production
pyroscope.write "production" {
    endpoint {
        url = "http://pyroscope-production:4041"
    }
    external_labels = {
        "env" = "production",
    }
}

Note
This configuration will duplicate the received profiles and send a copy to each configured pyroscope.write component.

Another approach is to configure multiple receivers with multiple destinations:

pyroscope.receive_http "staging" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9998
    }
    forward_to = [pyroscope.write.staging.receiver]
}

pyroscope.receive_http "production" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9999
    }
    forward_to = [pyroscope.write.production.receiver]
}

// Send profiles to staging
pyroscope.write "staging" {
    endpoint {
        url = "http://pyroscope-staging:4040"
    }
    external_labels = {
        "env" = "staging",
    }
}

// Send profiles to production
pyroscope.write "production" {
    endpoint {
        url = "http://pyroscope-production:4041"
    }
    external_labels = {
        "env" = "production",
    }
}

For more information about component configuration options, refer to:

pyroscope.receive_http documentation
pyroscope.write documentation

Set up Go profiling in pull mode

Wed, 08 Apr 2026 14:38:28 +0000

Set up Go profiling in pull mode

In pull mode, the collector, Grafana Alloy, periodically retrieves profiles from Golang applications, specifically targeting the /debug/pprof/* endpoints.

To set up Golang profiling in pull mode, you need to:

Expose pprof endpoints.
Install a collector, such as Grafana Alloy.
Prepare the collector’s configuration file.
Start the collector.

Expose pprof endpoints

Ensure your Golang application exposes pprof endpoints.

Get godeltaprof package

go get github.com/grafana/pyroscope-go/godeltaprof@latest

Import net/http/pprof and godeltaprof/http/pprof packages at the start of your application.
Go
```
import _ "net/http/pprof"
import _ "github.com/grafana/pyroscope-go/godeltaprof/http/pprof"
```

Install the collector

You can use the sample Alloy collector configuration file to send data to Pyroscope.

To install Alloy, refer to Grafana Alloy installation.

Prepare the collector configuration file

In the Alloy configuration file, you need to add at least two blocks: pyroscope.write and pyroscope.scrape.

Add pyroscope.write block.

pyroscope.write "write_job_name" {
        endpoint {
                url = "http://localhost:4040"
        }
}

Add pyroscope.scrape block.

pyroscope.scrape "scrape_job_name" {
        targets    = [{"__address__" = "localhost:4040", "service_name" = "example_service"}]
        forward_to = [pyroscope.write.write_job_name.receiver]

        profiling_config {
                profile.process_cpu {
                        enabled = true
                }

                profile.godeltaprof_memory {
                        enabled = true
                }

                profile.memory { // disable memory, use godeltaprof_memory instead
                        enabled = false
                }

                profile.godeltaprof_mutex {
                        enabled = true
                }

                profile.mutex { // disable mutex, use godeltaprof_mutex instead
                        enabled = false
                }

                profile.godeltaprof_block {
                        enabled = true
                }

                profile.block { // disable block, use godeltaprof_block instead
                        enabled = false
                }

                profile.goroutine {
                        enabled = true
                }
        }
}

Save the changes to the file.

Start the collector

Start a local Pyroscope instance for testing purposes:
Bash
```
docker run -p 4040:4040 grafana/pyroscope
```
To start Alloy v1.2 and later: Replace configuration.alloy with your configuration filename:
alloy run configuration.alloy
Open a browser to http://localhost:4040. The page should list profiles.

Examples

Send data to Grafana Cloud

Your Grafana Cloud URL, username, and password can be found on the “Details Page” for Pyroscope from your stack on grafana.com. On this same page, create a token and use it as the Basic authentication password.

pyroscope.write "write_job_name" {
        endpoint {
                url = "<Grafana Cloud URL>"

                basic_auth {
                        username = "<Grafana Cloud User>"
                        password = "<Grafana Cloud Password>"
                }
        }

}

Discover Kubernetes targets

Select all pods

discovery.kubernetes "all_pods" {
        role = "pod"
}

Drop not running pods, create namespace, pod, node and container labels. Compose service_name label based on namespace and container labels. Select only services matching regex pattern (ns1/.*)|(ns2/container-.*0).


discovery.relabel "specific_pods" {
        targets = discovery.kubernetes.all_pods.targets

        rule {
                action        = "drop"
                regex         = "Succeeded|Failed|Completed"
                source_labels = ["__meta_kubernetes_pod_phase"]
        }

        rule {
                action        = "replace"
                source_labels = ["__meta_kubernetes_namespace"]
                target_label  = "namespace"
        }

        rule {
                action        = "replace"
                source_labels = ["__meta_kubernetes_pod_name"]
                target_label  = "pod"
        }

        rule {
                action        = "replace"
                source_labels = ["__meta_kubernetes_node_name"]
                target_label  = "node"
        }

        rule {
                action        = "replace"
                source_labels = ["__meta_kubernetes_pod_container_name"]
                target_label  = "container"
        }

        rule {
                action        = "replace"
                regex         = "(.*)@(.*)"
                replacement   = "${1}/${2}"
                separator     = "@"
                source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
                target_label  = "service_name"
        }

        rule {
                action        = "keep"
                regex         = "(ns1/.*)|(ns2/container-.*0)"
                source_labels = ["service_name"]
        }
}

Use discovery.relabel.specific_pods.output as a target for pyroscope.scrape block.

    pyroscope.scrape "scrape_job_name" {
            targets    = discovery.relabel.specific_pods.output
            ...
    }

Exposing pprof endpoints

If you don’t use http.DefaultServeMux, you can register /debug/pprof/* handlers to your own http.ServeMux:

var mux *http.ServeMux
mux.Handle("/debug/pprof/", http.DefaultServeMux)

Or, if you use gorilla/mux:

var router *mux.Router
router.PathPrefix("/debug/pprof").Handler(http.DefaultServeMux)

References

Set up Java profiling using Grafana Alloy

Wed, 08 Apr 2026 14:38:28 +0000

Set up Java profiling using Grafana Alloy

Grafana Alloy supports Java profiling. The collector configuration file is composed of components that are used to collect, transform, and send data. The Alloy configuration files use the Alloy configuration syntax.

Configure the components

The pyroscope.java component is used to continuously profile Java processes running on the local Linux OS using async-profiler.

pyroscope.java "java" {
  profiling_config {
    interval = "15s"
    alloc = "512k"
    cpu = true
    event = "wall"
    per_thread = true
    lock = "10ms"
    sample_rate = 100
  }
  forward_to = [pyroscope.write.endpoint.receiver]
  targets = discovery.relabel.java.output
}

Using the targets argument, you can specify which processes and containers to profile on the machine.

The targets can be from discovery.process component. You can use discovery.process join argument to join process targets with extra discoveries such as discovery.kubernetes, discovery.docker, and discovery.dockerswarm. You can use the discovery.relabel component to relabel discovered targets and set your own labels. For more information, refer to the Components documentation.

The forward_to parameter should point to a pyroscope.write component to send the collected profiles to your Pyroscope Server or Grafana Cloud.

Name	Type	Description	Default	Required
`targets`	`list(map(string))`	List of java process targets to profile.		yes
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send collected profiles to.		yes
`tmp_dir`	`string`	Temporary directory to store async-profiler.	`/tmp`	no

The special label __process_pid__ must always be present in each target of targets and corresponds to the PID of the process to profile.

The special label service_name is required and must always be present. If service_name isn’t specified, pyroscope.java attempts to infer it from discovery meta labels. If service_name isn’t specified and couldn’t be inferred, then it’s set to unspecified.

The profiling_config block describes how async-profiler is invoked.

It supports the following arguments:

Name	Type	Description	Default	Required
`interval`	`duration`	How frequently to collect profiles from the targets.	“60s”	no
`cpu`	`bool`	A flag to enable CPU profiling, using `itimer` async-profiler event.	true	no
`event`	`string`	Sets the CPU profiling event. Can be one of `itimer`, `cpu`, or `wall`.	“itimer”	no
`sample_rate`	`int`	CPU profiling sample rate. It’s converted from Hz to interval and passed as `-i` arg to async-profiler.	100	no
`alloc`	`string`	Allocation profiling sampling configuration It’s passed as `--alloc` arg to async-profiler.	“512k”	no
`lock`	`string`	Lock profiling sampling configuration. It’s passed as `--lock` arg to async-profiler.	“10ms”	no
`per_thread`	`bool`	Enables async-profiler per-thread mode (`-t`). Recommended when using wall event profiling.	false	no

For more information on async-profiler configuration, see profiler-options.

Set privileges for the collector

You must run the collector, such Alloy, as root and inside host pid namespace for the pyroscope.java and discover.process components to work.

Start the collector

To start Grafana Alloy v1.2 and later, replace configuration.alloy with your configuration filename:

alloy run configuration.alloy

To start Grafana Alloy v1.0 or 1.1, replace configuration.alloy with your configuration file name:

alloy run --stability.level=public-preview configuration.alloy

The stability.level option is required for pyroscope.scrape with Alloy v1.0 or v1.1. For more information about stability.level, refer to the run command documentation.

Send data to Grafana Cloud Profiles

When sending to Grafana Cloud Profiles, you can use the following pyroscope.write component configuration which uses environment variables.

Ensure that you have appropriately configured the GC_URL, GC_USER, and GC_PASSWORD environment variables.

pyroscope.write "endpoint" {
    endpoint {
        basic_auth {
            password = env("GC_PASSWORD")
            username = env("GC_USER")
        }
        url = env("GC_URL")
    }
}

Examples

For more robust examples, refer to the Grafana Alloy and Agent Auto-instrumentation examples in the Pyroscope repository.

Profiling local process

discovery.process "all" {
}

discovery.relabel "java" {
    targets = discovery.process.all.targets
    // Filter only java processes
    rule {
        source_labels = ["__meta_process_exe"]
        action = "keep"
        regex = ".*/java$"
    }
    // Filter processes. For example: only processes with command line containing "FastSlow"
    rule {
        source_labels = ["__meta_process_commandline"]
        regex = "java FastSlow"
        action = "keep"
    }
    // Provide a service name for the process, otherwise it will be unspecified.
    rule {
        action = "replace"
        target_label = "service_name"
        replacement = "java-fast-slow"
    }
}

pyroscope.java "java" {
  forward_to = [pyroscope.write.example.receiver]
  targets = discovery.relabel.java.output
}

pyroscope.write "example" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Profiling Docker containers

For a working example, refer to Java profiling via auto-instrumentation example in Docker.

discovery.docker "local_containers" {
  host = "unix:///var/run/docker.sock"
}

discovery.process "all" {
  join = discovery.docker.local_containers.targets
}

discovery.relabel "java" {
    targets = discovery.process.all.targets
    // Filter only java processes
    rule {
        source_labels = ["__meta_process_exe"]
        action = "keep"
        regex = ".*/java$"
    }
    // Filter only needed containers
    rule {
        source_labels = ["__meta_docker_container_name"]
        regex = ".*suspicious_pascal"
        action = "keep"
    }
    // Provide a service name for the process, otherwise it will default to the value of __meta_docker_container_name label.
    rule {
        action = "replace"
        target_label = "service_name"
        replacement = "java-fast-slow"
    }
}

pyroscope.java "java" {
  forward_to = [pyroscope.write.example.receiver]
  targets = discovery.relabel.java.output
}

pyroscope.write "example" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Profiling Kubernetes pods

For a working example, refer to Grafana Alloy Java profiling via auto-instrumentation with Kubernetes.

discovery.kubernetes "local_pods" {
  selectors {
    field = "spec.nodeName=" + env("HOSTNAME")
    role = "pod"
  }
  role = "pod"
}

discovery.process "all" {
  join = discovery.kubernetes.local_pods.targets
}

discovery.relabel "java_pods" {
  targets = discovery.process.all.targets
  // Filter only java processes
  rule {
    source_labels = ["__meta_process_exe"]
    action = "keep"
    regex = ".*/java$"
  }
  rule {
    action = "drop"
    regex = "Succeeded|Failed|Completed"
    source_labels = ["__meta_kubernetes_pod_phase"]
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_namespace"]
    target_label = "namespace"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label = "pod"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label = "node"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label = "container"
  }
  // Provide arbitrary service_name label, otherwise it will be inferred from discovery labels automatically
  rule {
    action = "replace"
    regex = "(.*)@(.*)"
    replacement = "java/${1}/${2}"
    separator = "@"
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    target_label = "service_name"
  }
  // Filter only needed services
  rule {
    action = "keep"
    regex = "(java/ns1/.*)|(java/ns2/container-.*0)"
    source_labels = ["service_name"]
  }
}

pyroscope.java "java" {
  forward_to = [pyroscope.write.example.receiver]
  targets = discovery.relabel.java_pods.output
}

pyroscope.write "example" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

References

For more information:

Examples

Set up profiling with eBPF with Grafana Alloy

Wed, 08 Apr 2026 14:38:28 +0000

Set up profiling with eBPF with Grafana Alloy

eBPF is an advanced technology embedded into the Linux kernel. It stands for enhanced Berkeley Packet Filter and revolutionizes the capability to run sandboxed code safely within the kernel space. This technology serves multiple use cases, such as networking, security, and performance monitoring without the need to alter kernel code or load additional modules.

Benefits and tradeoffs of using eBPF for continuous profiling

When it comes to application profiling, eBPF offers high efficiency and minimal performance overhead. eBPF enables the dynamic insertion of powerful monitoring code into live production systems. By leveraging eBPF, developers can gain insights into application behavior, track resource usage, and detect bottlenecks in a way that traditional profiling tools cannot match. eBPF’s low overhead and fine-grained data collection make it an ideal choice for continuous, real-time profiling in performance-sensitive environments.

However, eBPF has some limitations that make it unsuitable for certain use cases:

It isn’t a good fit for profiling applications that arn’t written in a supported language.
It can’t be used to profile applications that aren’t running on Linux.
It doesn’t support all profile types such as memory and contention/lock profiling.
eBPF requires root access to the host machine, which can be a problem in some environments.

Supported languages

The eBPF profiler collects CPU profiles. Natively compiled languages like C/C++, Go, Rust, and Zig are supported. Frame pointers are not required — the profiler uses .eh_frame data for unwinding. Refer to Troubleshooting unknown symbols for additional requirements and information.

The following high-level languages are also supported: Java (Hotspot JVM), .NET, Python, Ruby, PHP, Node.js, and Perl. Each high-level language can be individually enabled or disabled in the pyroscope.ebpf Alloy component configuration.

eBPF using Alloy

Grafana Alloy is a lightweight, all-in-one collector that can collect, transform, and ship observability data. For profiling, you can configure Alloy to collect eBPF profiles and send them to Pyroscope.

This section contains instructions for installing and configuring Alloy to collect eBPF profiles. For more information about Alloy itself, refer to the Alloy documentation.

Sampling scrape targets

Wed, 08 Apr 2026 14:38:28 +0000

Sampling scrape targets

Applications often have many instances deployed. While Pyroscope is designed to handle large amounts of profiling data, you may want only a subset of the application’s instances to be scraped.

For example, the volume of profiling data your application generates may make it unreasonable to profile every instance, or you might be targeting cost-reduction.

Through configuration of Grafana Alloy collector, Pyroscope can sample scrape targets.

Before you begin

Make sure you understand how to configure the collector to scrape targets and are familiar with the component configuration language. Alloy configuration files use the Alloy configuration syntax.

Configuration

The hashmod action and the modulus argument are used in conjunction to enable sampling behavior by sharding one or more labels. To read further on these concepts, refer to rule block documentation. In short, hashmod performs an MD5 hash on the source labels and modulus performs a modulus operation on the output.

The sample size can be modified by changing the value of modulus in the hashmod action and the regex argument in the keep action. The modulus value defines the number of shards, while the regex value selects a subset of the shards.

Note
Choose your source label(s) for the hashmod action carefully. They must uniquely define each scrape target or hashmod won’t be able to shard the targets uniformly.

For example, consider an application deployed on Kubernetes with 100 pod replicas, all uniquely identified by the label pod_hash. The following configuration is set to sample 15% of the pods:

discovery.kubernetes "profile_pods" {
  role = "pod"
}

discovery.relabel "profile_pods" {
  targets = concat(discovery.kubernetes.profile_pods.targets)

  // Other rule blocks ...

  rule {
    action        = "hashmod"
    source_labels = ["pod_hash"]
    modulus       = 100
    target_label  = "__tmp_hashmod"
  }

  rule {
    action        = "keep"
    source_labels = ["__tmp_hashmod"]
    regex         = "^([0-9]|1[0-4])$"
  }

  // Other rule blocks ...
}

Considerations

This strategy doesn’t guarantee precise sampling. Due to its reliance on an MD5 hash, there isn’t a perfectly uniform distribution of scrape targets into shards. Larger numbers of scrape targets yield increasingly accurate sampling.

Keep in mind, if the label hashed is deterministic, you see deterministic sharding and thereby deterministic sampling of scrape targets. Similarly, if the label hashed is non-deterministic, you see scrape targets sampled in a non-deterministic fashion.