Pyroscope architecture on Grafana Labs

About the Pyroscope architecture

Wed, 08 Apr 2026 14:38:28 +0000

About the Pyroscope architecture

Pyroscope has a microservices-based architecture. The system has multiple horizontally scalable microservices that can run separately and in parallel. Pyroscope microservices are called components.

Pyroscope’s design compiles the code for all components into a single binary. The -target parameter controls which component(s) that single binary will behave as. For those looking for a simple way to get started, Pyroscope can also be run in monolithic mode, with all components running simultaneously in one process. For more information, refer to Deployment modes.

Pyroscope components

Most components are stateless and do not require any data persisted between process restarts. Some components are stateful and rely on non-volatile storage to prevent data loss between process restarts. For details about each component, see its page in Components.

The write path

Ingesters receive incoming profiles from the distributors. Each push request belongs to a tenant, and the ingester appends the received profiles to the specific per-tenant Pyroscope database that is stored on the local disk.

The per-tenant Pyroscope database is lazily created in each ingester as soon as the first profiles are received for that tenant.

The in-memory profiles are periodically flushed to disk and new block is created.

For more information, refer to Ingester.

Series sharding and replication

By default, each profile series is replicated to three ingesters, and each ingester writes its own block to the long-term storage. The Compactor merges blocks from multiple ingesters into a single block, and removes duplicate samples. Blocks compaction significantly reduces storage utilization.

The read path

Queries coming into Pyroscope arrive at query-frontend component which is responsible for accelerating queries and dispatching them to the query-scheduler.

The query-scheduler maintains a queue of queries and ensures that each tenant’s queries are fairly executed.

The queriers act as workers, pulling queries from the queue in the query-scheduler. The queriers connect to the ingesters to fetch all the data needed to execute a query. For more information about how the query is executed, refer to querier.

Depending on the time window selected, the querier involves ingesters for recent data and store-gateways for data from long-term storage.

Long-term storage

The Pyroscope storage format is described in detail in on the block format page. The Pyroscope storage format stores each tenant’s profiles into their own on-disk block. Each on-disk block directory contains an index file, a file containing metadata, and the Parquet tables.

Pyroscope requires any of the following object stores for the block files:

For more information, refer to configure object storage and configure disk storage.

Pyroscope deployment modes

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope deployment modes

You can deploy Pyroscope in one of two modes:

Monolithic mode: In this mode all components run in a single process and is meant to be used when you only need one pyroscope instance as multiple instances will not share information with each other
Microservices mode: In this mode in this mode as you scale out the number of instances, they will share a singular backend for storage and querying

The deployment mode is determined by the -target parameter, which you can set via CLI flag or YAML configuration.

Monolithic mode

The monolithic mode runs all required components in a single process and is the default mode of operation, which you can set by specifying -target=all. Monolithic mode is the simplest way to deploy Pyroscope and is useful if you want to get started quickly or want to work with Pyroscope in a development environment. To see the list of components that run when -target is set to all, run Pyroscope with the -modules flag:

./pyroscope -modules

Microservices mode

In microservices mode, components are deployed in distinct processes. Scaling is per component, which allows for greater flexibility in scaling and more granular failure domains. Microservices mode is the preferred method for a production deployment, but it is also the most complex.

In microservices mode, each Pyroscope process is invoked with its -target parameter set to a specific Pyroscope component (for example, -target=ingester or -target=distributor). To get a working Pyroscope instance, you must deploy every required component. For more information about each of the Pyroscope components, refer to Architecture.

If you are interested in deploying Pyroscope in microservices mode, we recommend that you use Kubernetes.

Pyroscope components

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope components

Pyroscope includes a set of components that interact to form a cluster.

Grafana Pyroscope bucket index

Wed, 08 Apr 2026 14:38:28 +0000

Grafana Pyroscope bucket index

The bucket index is a per-tenant file that contains the list of blocks and block deletion marks in the storage. The bucket index is stored in the backend object storage, is periodically updated by the compactor, and used by store-gateways to discover blocks in the storage.

Benefits

The store-gateway must have an almost¹ up-to-date view of the storage bucket, in order to find the right blocks to look up at query time and to load a block.

Because of this, they need to periodically scan the bucket to look for new blocks uploaded by ingesters or compactors, and blocks deleted (or marked for deletion) by compactors.

When the bucket index is enabled, store-gateways periodically look up the per-tenant bucket index instead of scanning the bucket via list objects operations.

This provides the following benefits:

Reduced number of API calls to the object storage by store-gateway
No “list objects” storage API calls performed by store-gateway

Structure of the index

The bucket-index.json.gz contains:

blocks
List of complete blocks of a tenant, including blocks marked for deletion. Partial blocks are excluded from the index.
block_deletion_marks
List of block deletion marks.
updated_at
A Unix timestamp, with precision measured in seconds, displays the last time index was updated and written to the storage.

How it gets updated

The compactor periodically scans the bucket and uploads an updated bucket index to the storage. You can configure the frequency with which the bucket index is updated via -compactor.cleanup-interval.

The use of the bucket index is optional, but the index is built and updated by the compactor even if -blocks-storage.bucket-store.bucket-index.enabled=false. This behavior ensures that the bucket index for any tenant exists and that query result consistency is guaranteed if a Grafana Pyroscope cluster operator enables the bucket index in a live cluster. The overhead introduced by keeping the bucket index updated is not significant.

How it’s used by the store-gateway

The store-gateway, at startup and periodically, fetches the bucket index for each tenant that belongs to its shard, and uses it as the source of truth for the blocks and deletion marks in the storage. This removes the need to periodically scan the bucket to discover blocks belonging to its shard.

Ingesters regularly add new blocks to the bucket as they offload data to long-term storage, and compactors subsequently compact these blocks and mark the original blocks for deletion. Actual deletion happens after the delay value that is associated with the parameter -compactor.deletion-delay. An attempt to fetch a deleted block will lead to failure of the query. Therefore, in this context, an almost up-to-date view is a view that’s outdated by less than the value of -compactor.deletion-delay. ↩︎

Pyroscope Block format

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope Block format

This document describes how Pyroscope stores the data in its blocks. Each block belongs to a single tenant and is identified by a unique ULID. Within the block there are multiple files:

A metadata file meta.json, which contains information about what the block contains, like the time range of the profiling data.
A TSDB index index.tsdb mapping the external labels to the profiles stored in the profiles table.
profiles.parquet parquet table that contains profiles.
symbols.symdb that contains symbolic information for the profiles stored in the block.

Data model

The data model within the block is fairly aligned to Google’s proto definition for the pprof wire format.

Profile series labels contain additional information gathered at ingestion time and can be used to select certain profiles. They are comparable to Prometheus/Loki labels and typical label names are namespace and pod to describe which workload the profiles are coming from.

Each profile ingested will be added into a new row in the profile table. If there are entries missing in the tables for the different models they are also inserted.

Pyroscope hash rings

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope hash rings

Hash rings are a distributed consistent hashing scheme and are widely used by Pyroscope for sharding and replication.

How the hash ring works in Pyroscope

The hash ring in Pyroscope is used to share work across several replicas of a component in a consistent way, so that any other component can decide which address to talk to. The workload or data to share is hashed first and the result of the hashing is used to find which ring member owns it.

Pyroscope uses the fnv32a hash function, which returns 32-bit unsigned integers so its value can be between 0 and (2^32)-1, inclusive. This value is called token and used as the ID of the data. The token determines the location on the hash ring deterministically. This allows independent determination of what instance of Pyroscope is the authoritative owner of any specific data.

For example, profiles are sharded across ingesters. The token of a given profile is computed by hashing all of the profile’s labels and the tenant ID: the result of which is an unsigned 32-bit integer within the space of the tokens. The ingester that owns that series is the instance that owns the range of the tokens, including the profile token.

To divide up set of possible tokens (2^32) across the available instances within the cluster, all of the running instances of a given Pyroscope component, such as the ingesters, join a hash ring. The hash ring is a data structure that splits the space of the tokens into multiple ranges, and assigns each range to a given Pyroscope ring member.

Upon startup, an instance generates random token values, and it registers them into the ring. The values that each instance registers determine which instance owns a given token. A token is owned by the instance that registered the smallest value that is higher than the token being looked up (by wrapping around zero when it reaches (2^32)-1)).

To replicate the data across multiple instances, Pyroscope finds the replicas by starting from the authoritative owner of the data and walking the ring clockwise. Data is replicated to the next instances found while walking the ring.

A practical example

To better understand how it works, take four ingesters and a tokens space between 0 and 9 as an example:

Ingester #1 is registered in the ring with the token 2
Ingester #2 is registered in the ring with the token 4
Ingester #3 is registered in the ring with the token 6
Ingester #4 is registered in the ring with the token 9

Pyroscope receives an incoming performance profile with labels {__name__="process_cpu", instance="1.1.1.1"}. It hashes the profile’s labels, and the result of the hashing function is the token 3.

To find which ingester owns token 3, Pyroscope looks up the token 3 in the ring and finds the ingester that is registered with the smallest token larger than 3. The ingester #2, which is registered with token 4, is the authoritative owner of the profile {__name__="process_cpu",instance="1.1.1.1"}.

With replication set to 3, Pyroscope replicates each profile to three ingesters. After finding the authoritative owner of the profile, Pyroscope continues to walk the ring clockwise to find the remaining two instances where the profile should be replicated. In the example that follows, the profile is replicated to the instances of Ingester #3 and Ingester #4.

Consistent hashing

The hash ring guarantees the property known as consistent hashing.

When an instance is added or removed from the ring, consistent hashing minimizes the number of tokens that are moved from one instance to another. On average, the number of tokens that need to move to a different instance is only n/m, where n is the total number of tokens (32-bit unsigned integer) and m is the number of instances that are registered in the ring.

Components that use the hash ring

There are several Pyroscope components that need a hash ring. Each of the following components builds an independent hash ring:

Ingesters shard and replicate series.
Distributors enforce rate limits.

How the hash ring is shared between Pyroscope instances

Hash ring data structures need to be shared between Pyroscope instances. To propagate changes to the hash ring, Pyroscope uses a key-value store. The key-value store is required and can be configured independently for the hash rings of different components.

For more information, see the memberlist documentation.

Features that are built using the hash ring

Pyroscope primarily uses the hash ring for sharding and replication. Features that are built using the hash ring:

Service discovery: Instances can discover each other looking up who is registered in the ring.
Heartbeats: Instances periodically send a heartbeat to the ring to signal they’re up and running. An instance is considered unhealthy if it misses the heartbeat for some period of time.

Pyroscope memberlist and gossip protocol

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope memberlist and gossip protocol

Memberlist is a Go library that manages cluster membership, node failure detection, and message passing using a gossip-based protocol. Memberlist is eventually consistent and network partitions are partially tolerated by attempting to communicate to potentially dead nodes through multiple routes.

Pyroscope uses memberlist to implement the hash ring data structures between instances.

Each instance maintains a copy of the hash rings. Each Pyroscope instance updates a hash ring locally and uses memberlist to propagate the changes to other instances. Updates generated locally and updates received from other instances are merged together to form the current state of the ring on the instance.

To configure memberlist, refer to configuring memberlist.

How memberlist propagates hash ring changes

When using a memberlist-based KV store, every Pyroscope instance propagates the hash ring data structures to other instances using the following techniques:

Propagating only the differences introduced in recent changes.
Propagating the full hash ring data structure.

Every -memberlist.gossip-interval an instance randomly selects a subset of all Pyroscope cluster instances configured by -memberlist.gossip-nodes and sends the latest changes to the selected instances. This operation is performed frequently and it’s the primary technique used to propagate changes.

In addition, every -memberlist.pullpush-interval an instance randomly selects another instance in the Pyroscope cluster and transfers the full content of the KV store, including all hash rings (unless -memberlist.pullpush-interval is zero, which disables this behavior). After this operation is complete, the two instances have the same content as the KV store. This operation is computationally more expensive, and as a result, it’s performed less frequently. The operation ensures that the hash rings periodically reconcile to a common state.