This is documentation for the next version of Grafana Alloy Documentation. For the latest stable release, go to the latest version.
pyroscope.receive_http
pyroscope.receive_http
receives profiles over HTTP and forwards them to pyroscope.*
components capable of receiving profiles.
The HTTP API exposed is compatible with both the Pyroscope HTTP ingest API and the pushv1.PusherService Connect API.
This allows pyroscope.receive_http
to act as a proxy for Pyroscope profiles, enabling flexible routing and distribution of profile data.
Usage
pyroscope.receive_http "<LABEL>" {
http {
listen_address = "<LISTEN_ADDRESS>"
listen_port = "<PORT>"
}
forward_to = <RECEIVER_LIST>
}
The component starts an HTTP server supporting the following endpoints:
POST /ingest
: Send profiles to the component, which forwards them to the receivers configured in theforward_to
argument. The request format must match the format of the Pyroscope ingest API.POST /push.v1.PusherService/Push
: Send profiles to the component, which forwards them to the receivers configured in theforward_to
argument. The request format must match the format of the Pyroscope pushv1.PusherService Connect API.
Arguments
You can use the following argument with pyroscope.receive_http
:
Name | Type | Description | Default | Required |
---|---|---|---|---|
forward_to | list(ProfilesReceiver) | List of receivers to send profiles to. | yes |
Blocks
You can use the following blocks with pyroscope.receive_http
:
Name | Description | Required |
---|---|---|
http | Configures the HTTP server that receives requests. | no |
http > tls | Configures TLS for the HTTP server. | no |
The > symbol indicates deeper levels of nesting.
For example, http
> tls
refers to a tls
block defined inside an http
block.
http
The http
block configures the HTTP server.
You can use the following arguments to configure the http
block. Any omitted fields take their default values.
Name | Type | Description | Default | Required |
---|---|---|---|---|
conn_limit | int | Maximum number of simultaneous HTTP connections. Defaults to no limit. | 0 | no |
listen_address | string | Network address on which the server listens for new connections. Defaults to accepting all incoming connections. | "" | no |
listen_port | int | Port number on which the server listens for new connections. | 8080 | no |
server_idle_timeout | duration | Idle timeout for HTTP server. | "120s" | no |
server_read_timeout | duration | Read timeout for HTTP server. | "30s" | no |
server_write_timeout | duration | Write timeout for HTTP server. | "30s" | no |
tls
The tls
block configures TLS for the HTTP server.
Name | Type | Description | Default | Required |
---|---|---|---|---|
cert_pem | string | PEM data of the server TLS certificate. | "" | no |
cert_file | string | Path to the server TLS certificate on disk. | "" | no |
client_auth_type | string | Client authentication to use. | "NoClientCert" | no |
client_ca_file | string | Path to the client CA file on disk to validate requests against. | "" | no |
client_ca_pem | string | PEM data of the client CA to validate requests against. | "" | no |
key_file | string | Path to the server TLS key on disk. | "" | no |
key_pem | secret | PEM data of the server TLS key. | "" | no |
The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:
cert_pem
andcert_file
key_pem
andkey_file
When configuring client authentication, both the client certificate (using cert_pem
or cert_file
) and the client key (using key_pem
or key_file
) must be provided.
Exported fields
pyroscope.receive_http
doesn’t export any fields.
Component health
pyroscope.receive_http
is reported as unhealthy if it’s given an invalid configuration.
Debug metrics
pyroscope_receive_http_tcp_connections
(gauge): Current number of accepted TCP connections.
pyroscope_receive_http_tcp_connections_limit
(gauge): The maximum number of TCP connections that the component can accept. A value of 0 means no limit.
Troubleshoot
Connection limit errors
When using pyroscope.write
to push profiles to a pyroscope.receive_http
component, you may encounter errors like:
"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"
This typically indicates that the receiving component has reached its TCP connection limit.
To resolve connection limit errors, first diagnose the issue, then apply one of the solutions.
Diagnose connection limit issues
Check the connection metrics on the
pyroscope.receive_http
component:pyroscope_receive_http_tcp_connections
: Current number of accepted TCP connectionspyroscope_receive_http_tcp_connections_limit
: Maximum number of TCP connections allowed
If the current connections are approaching or at the limit, you need to take action.
Solution 1: Increase the connection limit
To increase the connection limit, increase the conn_limit
parameter in the pyroscope.receive_http
configuration:
pyroscope.receive_http "example" {
http {
conn_limit = 32768 // Increase from default 16384
// ... other settings
}
// ... rest of configuration
}
Solution 2: Horizontal scaling
To distribute the connection load across multiple receivers, deploy multiple instances of pyroscope.receive_http
behind a load balancer.
Timeout chain issues
When chaining multiple Pyroscope components such as pyroscope.write
to pyroscope.receive_http
to another pyroscope.write
, you may encounter timeout issues that prevent retries:
"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"
Understand the problem
This issue occurs when:
- The first
pyroscope.write
component sends profiles topyroscope.receive_http
. - The
pyroscope.receive_http
component forwards profiles to anotherpyroscope.write
component. - Both
pyroscope.write
components have the same defaultremote_timeout
of 10 seconds. - The request context passes from the first
pyroscope.write
throughpyroscope.receive_http
to the secondpyroscope.write
, maintaining the original 10-second deadline. - If the second
pyroscope.write
component’s downstream request takes the full 10 seconds due to a broken TCP idle connection, there’s no time left for retries.
To resolve timeout chain issues, apply one of the following solutions.
Solution 1: Increase timeout on the first pyroscope.write
To provide buffer time for retries, increase the remote_timeout
on the initial pyroscope.write
component:
pyroscope.write "w1" {
endpoint {
url = "http://pyroscope-receiver:8080"
remote_timeout = "30s" // Increased from default 10s
// ... other settings
}
// ... rest of configuration
}
Solution 2: Decrease timeout on the downstream pyroscope.write
To ensure faster failures and allow time for retries, reduce the remote_timeout
on the downstream pyroscope.write
component:
pyroscope.write "w2" {
endpoint {
url = "http://pyroscope-backend:4040"
remote_timeout = "3s" // Reduced from default 10s
// ... other settings
}
// ... rest of configuration
}
Important considerations
- Normal latency: Pyroscope servers with the new architecture typically have 500-1000ms average latency for requests
- Timeout buffer: Always leave sufficient buffer time for retries when chaining components
- Retry configuration: Consider adjusting
max_backoff_retries
and backoff periods alongside timeout values - Monitoring: Monitor the
pyroscope_write_latency
metric to understand actual request latencies and adjust timeouts accordingly
Example
This example creates a pyroscope.receive_http
component, which starts an HTTP server listening on 0.0.0.0
and port 9999
.
The server receives profiles and forwards them to multiple pyroscope.write
components, which write these profiles to different HTTP endpoints.
// Receives profiles over HTTP
pyroscope.receive_http "default" {
http {
listen_address = "0.0.0.0"
listen_port = 9999
}
forward_to = [pyroscope.write.staging.receiver, pyroscope.write.production.receiver]
}
// Send profiles to a staging Pyroscope instance
pyroscope.write "staging" {
endpoint {
url = "http://pyroscope-staging:4040"
}
}
// Send profiles to a production Pyroscope instance
pyroscope.write "production" {
endpoint {
url = "http://pyroscope-production:4040"
}
}
Note
This example demonstrates forwarding to multiple
pyroscope.write
components. This configuration duplicates the received profiles and sends a copy to each configuredpyroscope.write
component.
You can also create multiple pyroscope.receive_http
components with different configurations to listen on different addresses or ports as needed.
This flexibility allows you to design a setup that best fits your infrastructure and profile routing requirements.
Compatible components
pyroscope.receive_http
can accept arguments from the following components:
- Components that export Pyroscope
ProfilesReceiver
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.