Menu

This is documentation for the next version of Grafana Alloy Documentation. For the latest stable release, go to the latest version.

General availability (GA) Open source

pyroscope.receive_http

pyroscope.receive_http receives profiles over HTTP and forwards them to pyroscope.* components capable of receiving profiles.

The HTTP API exposed is compatible with both the Pyroscope HTTP ingest API and the pushv1.PusherService Connect API. This allows pyroscope.receive_http to act as a proxy for Pyroscope profiles, enabling flexible routing and distribution of profile data.

Usage

alloy
pyroscope.receive_http "<LABEL>" {
  http {
    listen_address = "<LISTEN_ADDRESS>"
    listen_port = "<PORT>"
  }
  forward_to = <RECEIVER_LIST>
}

The component starts an HTTP server supporting the following endpoints:

  • POST /ingest: Send profiles to the component, which forwards them to the receivers configured in the forward_to argument. The request format must match the format of the Pyroscope ingest API.
  • POST /push.v1.PusherService/Push: Send profiles to the component, which forwards them to the receivers configured in the forward_to argument. The request format must match the format of the Pyroscope pushv1.PusherService Connect API.

Arguments

You can use the following argument with pyroscope.receive_http:

NameTypeDescriptionDefaultRequired
forward_tolist(ProfilesReceiver)List of receivers to send profiles to.yes

Blocks

You can use the following blocks with pyroscope.receive_http:

NameDescriptionRequired
httpConfigures the HTTP server that receives requests.no
http > tlsConfigures TLS for the HTTP server.no

The > symbol indicates deeper levels of nesting. For example, http > tls refers to a tls block defined inside an http block.

http

The http block configures the HTTP server.

You can use the following arguments to configure the http block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
conn_limitintMaximum number of simultaneous HTTP connections. Defaults to no limit.0no
listen_addressstringNetwork address on which the server listens for new connections. Defaults to accepting all incoming connections.""no
listen_portintPort number on which the server listens for new connections.8080no
server_idle_timeoutdurationIdle timeout for HTTP server."120s"no
server_read_timeoutdurationRead timeout for HTTP server."30s"no
server_write_timeoutdurationWrite timeout for HTTP server."30s"no

tls

The tls block configures TLS for the HTTP server.

NameTypeDescriptionDefaultRequired
cert_pemstringPEM data of the server TLS certificate.""no
cert_filestringPath to the server TLS certificate on disk.""no
client_auth_typestringClient authentication to use."NoClientCert"no
client_ca_filestringPath to the client CA file on disk to validate requests against.""no
client_ca_pemstringPEM data of the client CA to validate requests against.""no
key_filestringPath to the server TLS key on disk.""no
key_pemsecretPEM data of the server TLS key.""no

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

  • cert_pem and cert_file
  • key_pem and key_file

When configuring client authentication, both the client certificate (using cert_pem or cert_file) and the client key (using key_pem or key_file) must be provided.

Exported fields

pyroscope.receive_http doesn’t export any fields.

Component health

pyroscope.receive_http is reported as unhealthy if it’s given an invalid configuration.

Debug metrics

pyroscope_receive_http_tcp_connections (gauge): Current number of accepted TCP connections. pyroscope_receive_http_tcp_connections_limit (gauge): The maximum number of TCP connections that the component can accept. A value of 0 means no limit.

Troubleshoot

Connection limit errors

When using pyroscope.write to push profiles to a pyroscope.receive_http component, you may encounter errors like:

text
"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"

This typically indicates that the receiving component has reached its TCP connection limit.

To resolve connection limit errors, first diagnose the issue, then apply one of the solutions.

Diagnose connection limit issues

  1. Check the connection metrics on the pyroscope.receive_http component:

    • pyroscope_receive_http_tcp_connections: Current number of accepted TCP connections
    • pyroscope_receive_http_tcp_connections_limit: Maximum number of TCP connections allowed
  2. If the current connections are approaching or at the limit, you need to take action.

Solution 1: Increase the connection limit

To increase the connection limit, increase the conn_limit parameter in the pyroscope.receive_http configuration:

alloy
pyroscope.receive_http "example" {
  http {
    conn_limit = 32768  // Increase from default 16384
    // ... other settings
  }
  // ... rest of configuration
}

Solution 2: Horizontal scaling

To distribute the connection load across multiple receivers, deploy multiple instances of pyroscope.receive_http behind a load balancer.

Timeout chain issues

When chaining multiple Pyroscope components such as pyroscope.write to pyroscope.receive_http to another pyroscope.write, you may encounter timeout issues that prevent retries:

text
"failed to push to endpoint" err="deadline_exceeded: context deadline exceeded"

Understand the problem

This issue occurs when:

  1. The first pyroscope.write component sends profiles to pyroscope.receive_http.
  2. The pyroscope.receive_http component forwards profiles to another pyroscope.write component.
  3. Both pyroscope.write components have the same default remote_timeout of 10 seconds.
  4. The request context passes from the first pyroscope.write through pyroscope.receive_http to the second pyroscope.write, maintaining the original 10-second deadline.
  5. If the second pyroscope.write component’s downstream request takes the full 10 seconds due to a broken TCP idle connection, there’s no time left for retries.

To resolve timeout chain issues, apply one of the following solutions.

Solution 1: Increase timeout on the first pyroscope.write

To provide buffer time for retries, increase the remote_timeout on the initial pyroscope.write component:

alloy
pyroscope.write "w1" {
  endpoint {
    url = "http://pyroscope-receiver:8080"
    remote_timeout = "30s"  // Increased from default 10s
    // ... other settings
  }
  // ... rest of configuration
}

Solution 2: Decrease timeout on the downstream pyroscope.write

To ensure faster failures and allow time for retries, reduce the remote_timeout on the downstream pyroscope.write component:

alloy
pyroscope.write "w2" {
  endpoint {
    url = "http://pyroscope-backend:4040"
    remote_timeout = "3s"  // Reduced from default 10s
    // ... other settings
  }
  // ... rest of configuration
}

Important considerations

  • Normal latency: Pyroscope servers with the new architecture typically have 500-1000ms average latency for requests
  • Timeout buffer: Always leave sufficient buffer time for retries when chaining components
  • Retry configuration: Consider adjusting max_backoff_retries and backoff periods alongside timeout values
  • Monitoring: Monitor the pyroscope_write_latency metric to understand actual request latencies and adjust timeouts accordingly

Example

This example creates a pyroscope.receive_http component, which starts an HTTP server listening on 0.0.0.0 and port 9999. The server receives profiles and forwards them to multiple pyroscope.write components, which write these profiles to different HTTP endpoints.

alloy
// Receives profiles over HTTP
pyroscope.receive_http "default" {
  http {
    listen_address = "0.0.0.0"
    listen_port = 9999
  }
  forward_to = [pyroscope.write.staging.receiver, pyroscope.write.production.receiver]
}

// Send profiles to a staging Pyroscope instance
pyroscope.write "staging" {
  endpoint {
    url = "http://pyroscope-staging:4040"
  }
}

// Send profiles to a production Pyroscope instance
pyroscope.write "production" {
  endpoint {
    url = "http://pyroscope-production:4040"
  }
}

Note

This example demonstrates forwarding to multiple pyroscope.write components. This configuration duplicates the received profiles and sends a copy to each configured pyroscope.write component.

You can also create multiple pyroscope.receive_http components with different configurations to listen on different addresses or ports as needed. This flexibility allows you to design a setup that best fits your infrastructure and profile routing requirements.

Compatible components

pyroscope.receive_http can accept arguments from the following components:

Note

Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.