Documentation Grafana Alloy Troubleshoot Profile resource consumption

Profile Grafana Alloy resource consumption

Alloy is written in the Go programming language, which offers built-in support for profiling. Like other applications written in Go, you can profile Alloy by sending an HTTP request, which returns a pprof Go profile.

Once you have the pprof file, visualize it as a flame graph in Grafana Pyroscope. Alternatively, you could visualize it on Grafana Play or locally by using Go’s built-in pprof tool.

The port you use to send the HTTP request is controlled by the --server.http.listen-addr command line argument. It is set to 127.0.0.1:12345 by default.

Obtaining a single profile

Different types of HTTP requests will retrieve different profiles.

Memory consumption

Goroutine leaks often cause memory leaks. This is why obtaining a goroutine profile is usually necessary when investigating memory issues. For example:

bash Copy

curl localhost:12345/debug/pprof/heap -o heap.pprof
curl localhost:12345/debug/pprof/goroutine -o goroutine.pprof

It is often helpful to collect profiles both when memory usage is low and when it is high. You can compare the profiles, and it may be easier to identify what caused the memory consumption to increase.

CPU consumption

If you are experiencing high CPU consumption, you can collect a CPU profile:

bash Copy

curl http://localhost:12345/debug/pprof/profile?seconds=30 -o cpu.pprof

The ?seconds=30 part of the URL above means the profiling will continue for 30 seconds.

Continuous profiling

You don’t have to send manual curl commands each time you want to collect profiles. You can also profile continuously using the pyroscope components in Alloy.

If you have very few Alloy instances, you can even configure them to profile themselves. However, if you have a large cluster of collectors, it is best to set up Alloy instances whose sole job is to profile other Alloy instances.

The following is an example of an Alloy instance profiling itself:

alloy Copy

pyroscope.scrape "default" {
  targets    = [{"__address__" = "localhost:12345", "service_name"="alloy"}]
  forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
  endpoint {
    url = "https://profiles-prod-014.grafana.net"
    basic_auth {
      username = sys.env("PYROSCOPE_USERNAME")
      password = sys.env("PYROSCOPE_PASSWORD")
    }
  }
}

Expected resource consumption

Refer to Estimate resource usage for more information about the expected resource consumption in Alloy.

Alloy consumes an abnormally large amount of resources

If Alloy consumes an abnormally large amount of resources, you can open an issue in the Alloy repository. Attach your pprof files and your Alloy configuration file. Make sure you redact any secrets in the attachments.