How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons

Running Alloy as a single-instance sidecar is simple. Running it as a centralized gateway that absorbs the full telemetry stream of an enterprise platform—tens of millions of active series, terabytes of logs per day, and tens of thousands of trace spans per second—is a different challenge altogether. To get it right, you need deliberate capacity planning, honest load testing, and a monitoring setup that doesn't rely on the very thing you're testing.

As part of the Professional Services team here at Grafana Labs, we've seen this firsthand working with customers. In this post, we'll walk you through the best practices we follow to help them find success, and we'll do so using real, anonymized data from a recent engagement. 

We'll cover how we sized and load tested a production Alloy central collector deployment on Kubernetes, what the numbers looked like under real stress, and how the cluster behaves today handling the full production telemetry workload for a large enterprise platform. By the end, you should have a better sense for how you can create your own central gateway for collecting telemetry in Grafana Cloud.

Why a central gateway?

Before diving into numbers, it's worth explaining the pattern. In a central gateway setup, all telemetry from application teams—metrics, logs, and traces—flows to a shared Alloy fleet via OTLP or native Prometheus/Loki write protocols. Alloy buffers, processes, batches, and forwards everything to Grafana Cloud.

This gives you several things that per-team sidecar deployments struggle to provide:

• A single control plane: Auth, rate limiting, and routing in one place so application teams don't need to manage Grafana Cloud credentials
• Centralized buffering: Ensure a transient Grafana Cloud slowdown doesn't immediately cause data loss at the source
• Cost visibility: Configure the gateway to only accept telemetry data containing the label or attribute that is mandatory for cost-attribution
• Protocol normalization: Send OTLP, Prometheus Remote Write, or Loki-native writes; the gateway fans it all out correctly

The tradeoff is that this becomes a critical piece of infrastructure. When it struggles, everyone feels it. That's why you need to treat it like any other production service: capacity planning first, then load testing before go-live.

Knowing your numbers: capacity planning

The first step is understanding expected traffic volumes. In the customer's production environment we recently worked on, the expected ingestion looked like this:

The "expected" column reflects several onboarding waves from additional application teams over the following weeks. We needed to size for that headroom, not just the current baseline. Most large enterprises tend to follow a similar pattern of gradually adding teams, but it's important to prepare for that full baseline upfront because Kubernetes autoscaling is not fast enough to handle such a huge unplanned surge of data volume to your Alloy collector all at once.

Resource rules of thumb

Grafana publishes sizing guidance in the Alloy documentation. We recommend using those as a baseline when sizing your Alloy deployment. In our case, considering the expected telemetry data volumes, we planned for the following resource budgets:

• Metrics: ~187 GB / 7 CPU cores
• Logs: ~2.1 GB / 17.5 cores
• Traces: ~4 GiB / ~3 cores

Totals: ~195 GB memory / ~28 CPU cores

Rather than running a handful of large pods, we recommend distributing the workload across many smaller ones. Finding the right split will depend on your infrastructure, but smaller pods also make horizontal scaling easier and more reactive.

Here's how each pod was configured for our customer:

Notice there is no CPU limit—only a request. This is intentional. CPU throttling in Kubernetes is one of the most common sources of hidden latency in high-throughput workloads, and capping it artificially would create pressure that memory limits and autoscaling handle more gracefully anyway.

With ~195 GB total memory needed across 6 GiB pods, the expected pod count lands at around 33-35 under normal load. We configured the HPA (Horizontal Pod Autoscaling) with MinReplica: 30, MaxReplica: 100, targeting 70% CPU and 90% memory utilization. While the number of pods can vary based on your scenario, we believe that the target resource utilization provides the right stability for a smooth autoscaling of Alloy.

An important aspect to consider: resource utilization can vary widely, depending on the use case. Grafana Labs' metrics sizing guidance is for scraping metrics with Alloy and remote-writing them. It's not specific to the gateway model, where Alloy is receiving and remote-writing them. We haven't found a public benchmark that isolates this trade-off for the gateway model, so take this as directional rather than definitive. If you're sizing a receive-and-remote-write gateway, we'd lean on padding your memory budget beyond the formula's number and trust its CPU estimate more as-is.

Architecture: what the deployment looks like

Next, you need to start building your architecture so Alloy can collect your telemetry and send it to Grafana Cloud. Based on our experience with many customer engagements, we recommend placing your central collector behind an ingress controller in your Kubernetes environment. That way, all traffic—OTLP over HTTP/Protobuf, Prometheus Remote Write, and Loki HTTP push—terminates at the ingress and gets distributed across the Alloy pod fleet. 

Here's a look at the workflow we set up for our customer:

One detail worth calling out from the example above: the cluster's own monitoring runs on a separate path. We deployed the Kubernetes Monitoring Helm chart to scrape Alloy's own /metrics endpoint and ship that data directly to Grafana Cloud, without going through the central collector itself. If you do this in your own environment, Alloy's health metrics will never be at risk if the collector is under stress—you always have visibility into what's happening, even if the main pipeline is struggling.

Core Alloy configuration

The gateway configuration wires up receivers for all three protocols and exports them to Grafana Cloud.

These telemetry data pipelines do not handle any complex transformations. They just make sure that incoming data includes the mandatory labels/attributes that are established in the labelling strategy of the customer, with particular emphasis on the cost attribution label. 

Load testing: simulating real traffic

Before putting a production workload through any new piece of infrastructure, you need to know how it behaves under stress. We ran many load tests against a pre-production environment that mirrored the production configuration.

Tools

We used two tools:

• telemetrygen, the native OpenTelemetry tool for generating configurable spans, metrics, and logs over OTLP.
• k6 with a custom build for Prometheus Remote Write and Loki Write

To have a custom build of k6, you need xk6, the k6 extension development toolbox.

Both tools ran on 10 Amazon EC2 m5.2xlarge instances to distribute the load generation itself, with traffic entering the cluster via the same ingress controller as production. All instances were monitored and reported to Grafana Cloud by having Alloy running on each. We performed dozens of tests, generating volumes comparable to the expected production load, but we also included  tests that added more stress via higher volumes and prolonged ingestion. 

While we relied on EC2 instances, you can choose your preferred infrastructure where to run them, as long as you have the necessary resources the scripts would require to generate the needed volumes. 

The monitoring path ran separately—the Kubernetes Monitoring Helm chart scraped Alloy's own /metrics endpoint and pushed to Grafana Cloud directly, giving us real-time visibility into the collector's health without the test traffic interfering with those metrics.

What to monitor

When you're running Alloy as a gateway, the following metrics are the ones that matter most. We built dedicated dashboards around them, and we recommend you do the same, because it gives you a useful information on Alloy’s health and its operation:

Ingestion health

• otelcol_receiver_accepted_spans_total: Traces being accepted
• prometheus_remote_storage_bytes_total: Metrics throughput
• loki_write_sent_bytes_total: Log write request rate

Data loss signals

• otelcol_receiver_refused_spans_total: This should be zero; any non-zero value means spans are being dropped
• HTTP 4xx/5xx from otelcol_exporter_* metrics: Indicates issues reaching Grafana Cloud

Backpressure and buffering

• otelcol_exporter_queue_size: Watch this during traffic spikes to validate your queue config
• otelcol_exporter_queue_capacity: The ceiling your queue is configured to

Resource utilisation

• process_cpu_seconds_total and process_resident_memory_bytes on Alloy pods
• Standard Kubernetes cAdvisor metrics (Pod CPU/Memory usage): These feed the HPA

Ingress controller

• Any normal aggregation of successful requests vs. failed ones and their latency (RED metrics) can also be used to exclude ingress problems that sit in front of the Alloy service endpoints. You may need to tune it if any limits or performance bottlenecks are found.

We deliberately keep the monitoring scrape path separate from the gateway itself. If you use the gateway to ship its own health metrics, a struggling gateway can become invisible at exactly the moment you most need to see it.

Production: 60 pods, zero issues

The deployment has been running in production on the Kubernetes cluster since go-live. 

Alloy managed to scale successfully, from a baseline of 30 pods minimum to handle multiple customer deployments and ingestion waves.

At the time of writing, this deployment handles almost 17M active series (25M DPM), 20-30 MB/s of log ingestion, and up to 150 MB/s of traces ingestion. These volumes are even higher than what was initially scoped and planned. 

Here’s a snapshot of our main monitoring dashboard:

Overview

Accepted vs. refused spans

The HPA scales the fleet up and down smoothly as traffic fluctuates across the day, and the pod count over time tells a clean story: steady baseline, gradual ramp, no drama.

Lessons learned: a (sometimes painful) journey to stability

Getting here wasn't entirely smooth. A few things tripped us up before production, and understanding them might save you the same trouble.

The WAL will eat all your memory if you let it. Alloy's write-ahead log is one of its most important reliability features—it buffers data so an Alloy restart doesn't mean data loss. But under sustained high throughput, especially when Grafana Cloud is backpressuring, the WAL can grow faster than it's drained. Left unchecked, this turns into an OOM kill. We learned this the hard way during load testing.

GOMEMLIMIT is your last line of defence before an OOM kill. Go's runtime doesn't respect Kubernetes memory limits on its own—the container will grow until the kernel kills it. Setting GOMEMLIMIT at around 80% of your memory limit tells the Go garbage collector to start aggressively reclaiming memory before you hit the hard ceiling. In our setup, this acts as a soft-limit trigger:

If GOMEMLIMIT is invoked frequently, Grafana Cloud pressure causes CPU consumption to rise. That's actually a feature, not a bug—it means the 70% CPU HPA threshold kicks in and scales out more pods before memory becomes critical. The layered defence looks like this:

1. GOMEMLIMIT at 80% memory → triggers GC, CPU rises
2. HPA scales at 70% CPU → more pods, load distributed
3. HPA scales at 90% memory → final safety net
4. Pod memory limit at 100% → OOMKill (this should rarely be reached)

Size for your growth ceiling, not your current baseline. Our production traffic at go-live was well below the load test peaks. But we sized for the onboarding wave that was two weeks away, which meant the rollout was uneventful rather than a scramble.

Keep the monitoring path out of band. Running the Kubernetes Monitoring Helm chart with a direct path to Grafana Cloud was the right call. During the load tests, we had full visibility into Alloy's behavior at all times. This is especially important during outages. The last thing you want is your observability dead-zone to coincide with your gateway being unhealthy.

Test above your ceiling. Our test results were two to three times the expected production peak. That wasn't accidental. Knowing how the system behaves above the expected maximum tells you how much headroom you actually have and what the failure mode looks like.

retry_on_failure is not optional. Grafana Cloud rate limits are not hard limits, but during a burst they will backpressure. Without retry config on your exporters, that backpressure becomes data loss. With it, Alloy buffers and drains gracefully.

Match your HPA min-replica to your sizing estimate. Setting MinReplica: 30 (rather than letting the fleet scale all the way down to three to four pods) ensured there was always enough capacity to absorb a sudden ramp without a lag in scale-out. Cold-start latency on Kubernetes nodes is real; don't let it show up as a gap in your telemetry. The MaxReplica: 100 ceiling gives plenty of headroom for the growth waves we're planning for.

Keep an eye on cluster-autoscaler too. Scaling pods is nice but ultimately you may need to scale your cluster nodes as well (or at least be prepared to). Check if there are limits in place either explicit (at cluster definition level) or implicit (like available IPs in the cluster subnets).

What's next

The following enhancements fell outside our initial project scope—think of them as optimizations we flagged for future iterations. In some cases, we lacked a necessary building block (like KEDA), while in others, our existing configuration was performing so efficiently that immediate changes weren't warranted.

We're highlighting them now because intentional deferment is a vital component of any mature capacity planning strategy. If your central collector is currently stable, view these points as early warning indicators for future bottlenecks rather than a mandatory to-do list for today.

Smarter autoscaling with KEDA

CPU and memory are reasonable HPA signals, but they're lagging indicators. By the time the CPU has risen to 70%, the pipeline is already under pressure. A more direct signal is the exporter queue itself: if your OTLP exporter queue is filling up, work is accumulating faster than it's being sent—and that's the earliest warning you can get.

We're exploring KEDA to add queue-depth-based autoscaling alongside the existing HPA:

This would let the fleet scale out proactively—before CPU and memory climb—giving new pods time to warm up before the existing ones are saturated. KEDA is a prerequisite for this approach as well as its metrics source (for example, a dedicated, highly focused, in-cluster Prometheus instance), so it's on the roadmap once KEDA is established in our cluster.

Different deployments per signal

We use only one Alloy deployment to handle all telemetry data (metrics, logs, traces) but a “trivial” change, if supported by data, would be to split the incoming traffic to different Alloy deployments, each finely tuned for the type of data it handles. Different telemetry data processing can incur different patterns of resource consumption. 

Broker (Kafka/*MQ) 

An optional component of an ingestion layer is a data broker that decouples different parts of the pipeline. In our case it was not required but it’s worth mentioning since it’s a common option that has to be considered. With size it becomes a significant part to manage (and pay for) so careful consideration should be done.

Get started

If you're building a similar central gateway setup, the Alloy documentation is the best place to start—particularly the deployment guide and the clustering concepts for production scale. For managing large Alloy fleets, Fleet Management in Grafana Cloud lets you centralize config, monitor collector health, and roll out changes—all from one place.

Grafana Assistant is the easiest way to get started with metrics, logs, traces, dashboards, and more in Grafana Cloud. We have a generous forever-free tier and plans for every use case. Sign up for free now!