Menu
Grafana Cloud

Grafana Alloy collector reference

Use this reference if you want to configure Grafana Alloy instances without using the Kubernetes Monitoring configuration GUI or if you want to modify Alloy instances you have deployed.

Collectors are Grafana Alloy instances deployed by the Alloy Operator as Kubernetes workloads. This information covers collector options specific to the Kubernetes Monitoring Helm chart.

When you enable a collector, Alloy Operator creates a Kubernetes workload as either a DaemonSet, StatefulSet, or Deployment, with its own set of Pods running Alloy containers. Each collector uses a workload type appropriate for the type of telemetry it collects.

General configuration

Each collector is defined in its own section in the values file of the Kubernetes Monitoring Helm chart. To enable and configure a collector, follow this general format:

yaml
alloy-<collector name>:
  enabled: true # Enable deploying this collector

  alloy: # Settings related to the Alloy instance
    ...
  controller: # Settings related to the Alloy controller
    ...

Because collectors are deployed using the Alloy Operator, you can use any of the standard Alloy helm chart values. These values will be used when creating the Alloy instance.

Alloy Receiver

  • Pod name: <helm_release>-alloy-receiver-*
  • Default controller type: DaemonSet
  • Service name: <helm_release_name>-alloy-receiver

This collector creates an Alloy instance deployed as a DaemonSet to receive application metrics when the Application Observability feature is enabled.

For each receiver enabled, configure this collector to expose the corresponding ports on the Kubernetes service that is fronting the Pods. For example, to enable a receiver to collect Zipkin traces, add:

yaml
applicationObservability:
  enabled: true
  receivers:
    otlp:
      grpc:
        enabled: true
        port: 4317
      http:
        enabled: true
        port: 4318
---
alloy-receiver:
  enabled: true
  alloy:
    extraPorts:
      - name: otlp-grpc
        port: 4317
        targetPort: 4317
        protocol: TCP
      - name: otlp-http
        port: 4318
        targetPort: 4318
        protocol: TCP

Client endpoint configuration

You can configure endpoints inside or outside the Cluster.

Inside the Cluster

Applications inside the Kubernetes Cluster use the kubedns name to reference a particular receiver endpoint. For example:

yaml
endpoint: http://grafana-k8s-monitoring-alloy[.mynamespace.cluster.local]:4318

Outside the Cluster

To expose the receiver to applications outside the Cluster (for example, Frontend Observability), you can use different approaches depending on your setup. Load balancers are created by whatever controllers are installed on your Cluster. For the full list of options, refer to the Alloy chart values.

For example, to create a Network Load Balancer on Amazon Elastic Kubernetes Service (Amazon EKS) when using the AWS Load Balancer Controller, use this example:

yaml
alloy-receiver:
  alloy:
    service:
      type: LoadBalancer

To create an Application Load Balancer, use this example:

yaml
alloy-receiver:
  alloy:
    ingress:
      enabled: true
      path: /
      faroPort: 12347

You can also create additional services and ingress objects as needed if the Alloy Helm chart options don’t fit your needs. Consult your Kubernetes vendor documentation for details.

Istio/Service Mesh

Depending on your mesh configuration, you might need to do either of these:

  • Explicitly include the Grafana monitoring namespace as a member.
  • Declare the receiver as a backend of your application for traffic within the Cluster.

For traffic from outside the Cluster, it’s likely you need to set up an ingress gateway into your mesh. In any case, consult your mesh vendor for details.

Troubleshooting

Here are some troubleshooting tips related to configuring collectors.

Startup issues

Make sure your receiver Pods are up and running. Use this command to show you a list of Pods and associated states:

kubectl get pods -n <helm_release_namespace>

While you may have meta monitoring turned on (which exposes the Alloy Pod logs in Loki), this is not helpful when the alloy-logs receiver itself is faulty.

To troubleshoot receiver startup problems, inspect the Pod logs using the method you would for any Kubernetes workload. To watch the alloy-logs receiver Pods, use:

kubectl logs -f --tail 100 ds/grafana-k8s-monitoring-alloy-logs

Alloy debugger

You can apply standard Alloy troubleshooting strategies to each collector Pod specifically for Kubernetes.

  1. To access the Alloy UI on a collector Pod, forward the UI port to your local machine:

    bash
    kubectl port-forward grafana-k8s-monitoring-alloy-receiver 12345:12345
  2. Open your browser to http://localhost:12345

Scaling

Follow these instructions for appropriate scaling.

DaemonSets and Singleton instances

For collectors deployed as DaemonSets and Singleton instances, one Pod is deployed per Node. You cannot deploy more replicas with this type of controller. To scale the individual Pods, increase the resource requests and limits. Refer to Estimate Grafana Alloy resource usage to learn how to tune those parameters.

For example:

yaml
alloy-metrics:
  alloy:
    resources:
      requests: {}
      limits: {}

StatefulSets

For StatefulSet collectors, set the number of replicas in the alloy config section of the collector:

yaml
alloy-metrics:
  controller:
    replicas: 3

Autoscaling

Caution

Autoscalers can cause Cluster outages when not configured properly.

Alloy doesn’t provide autoscaling out of the box. However, depending on the type of deployment for the collector, you can use these Kubernetes autoscalers:

Set the target for the HPA or VPA to the collector deployment name.

Values reference

The following is a reference for for collector values.

Alloy Logs

KeyTypeDefaultDescription
alloy-logs.controller.typestring"daemonset"The type of controller to use for the Alloy Logs instance
alloy-logs.enabledboolfalseDeploys the Alloy instance for collecting log data
alloy-logs.extraConfigstring""Extra Alloy configuration to be added to the configuration file
alloy-logs.liveDebugging.enabledboolfalseEnables live debugging for the Alloy instance
alloy-logs.logging.formatstring"logfmt"Format to use for writing Alloy log lines
alloy-logs.logging.levelstring"info"Level at which Alloy log lines should be written
alloy-logs.remoteConfig.auth.passwordstring""The password to use for the remote config server
alloy-logs.remoteConfig.auth.passwordFromstring""Raw config for accessing the password
alloy-logs.remoteConfig.auth.passwordKeystring"password"The key for storing the username in the secret
alloy-logs.remoteConfig.auth.typestring"none"The type of authentication to use for the remote config server
alloy-logs.remoteConfig.auth.usernamestring""The username to use for the remote config server
alloy-logs.remoteConfig.auth.usernameFromstring""Raw config for accessing the username
alloy-logs.remoteConfig.auth.usernameKeystring"username"The key for storing the username in the secret
alloy-logs.remoteConfig.enabledboolfalseEnables fetching configuration from a remote config server
alloy-logs.remoteConfig.extraAttributesobject{}Attributes to be added to this collector when requesting configuration
alloy-logs.remoteConfig.pollFrequencystring"5m"The frequency at which to poll the remote config server for updates
alloy-logs.remoteConfig.proxyURLstring""The proxy URL to use of the remote config server
alloy-logs.remoteConfig.secret.createbooltrueWhether to create a secret for the remote config server
alloy-logs.remoteConfig.secret.embedboolfalseIf true, skips secret creation and embeds the credentials directly into the configuration
alloy-logs.remoteConfig.secret.namestring""The name of the secret to create
alloy-logs.remoteConfig.secret.namespacestring""The namespace for the secret
alloy-logs.remoteConfig.urlstring""The URL of the remote config server

Alloy Metrics

KeyTypeDefaultDescription
alloy-metrics.controller.replicasint1The number of replicas for the Alloy Metrics instance
alloy-metrics.controller.typestring"statefulset"The type of controller to use for the Alloy Metrics instance
alloy-metrics.enabledboolfalseDeploys the Alloy instance for collecting metrics
alloy-metrics.extraConfigstring""Extra Alloy configuration to be added to the configuration file
alloy-metrics.liveDebugging.enabledboolfalseEnables live debugging for the Alloy instance
alloy-metrics.logging.formatstring"logfmt"Format to use for writing Alloy log lines
alloy-metrics.logging.levelstring"info"Level at which Alloy log lines should be written
alloy-metrics.remoteConfig.auth.passwordstring""The password to use for the remote config server
alloy-metrics.remoteConfig.auth.passwordFromstring""Raw config for accessing the password
alloy-metrics.remoteConfig.auth.passwordKeystring"password"The key for storing the password in the secret
alloy-metrics.remoteConfig.auth.typestring"none"The type of authentication to use for the remote config server
alloy-metrics.remoteConfig.auth.usernamestring""The username to use for the remote config server
alloy-metrics.remoteConfig.auth.usernameFromstring""Raw config for accessing the password
alloy-metrics.remoteConfig.auth.usernameKeystring"username"The key for storing the username in the secret
alloy-metrics.remoteConfig.enabledboolfalseEnable fetching configuration from a remote config server
alloy-metrics.remoteConfig.extraAttributesobject{}Attributes to be added to this collector when requesting configuration
alloy-metrics.remoteConfig.pollFrequencystring"5m"The frequency at which to poll the remote config server for updates
alloy-metrics.remoteConfig.proxyURLstring""The proxy URL to use for the remote config server
alloy-metrics.remoteConfig.secret.createbooltrueWhether to create a secret for the remote config server
alloy-metrics.remoteConfig.secret.embedboolfalseIf true, skips secret creation and embed the credentials directly into the configuration
alloy-metrics.remoteConfig.secret.namestring""The name of the secret to create
alloy-metrics.remoteConfig.secret.namespacestring""The namespace for the secret
alloy-metrics.remoteConfig.urlstring""The URL of the remote config server

Alloy Profiles

KeyTypeDefaultDescription
alloy-profiles.controller.typestring"daemonset"The type of controller to use for the Alloy Profiles instance
alloy-profiles.enabledboolfalseDeploys the Alloy instance for gathering profiles
alloy-profiles.extraConfigstring""Extra Alloy configuration to be added to the configuration file
alloy-profiles.liveDebugging.enabledboolfalseEnables live debugging for the Alloy instance
alloy-profiles.logging.formatstring"logfmt"Format to use for writing Alloy log lines
alloy-profiles.logging.levelstring"info"Level at which Alloy log lines should be written
alloy-profiles.remoteConfig.auth.passwordstring""The password to use for the remote config server
alloy-profiles.remoteConfig.auth.passwordFromstring""Raw config for accessing the password
alloy-profiles.remoteConfig.auth.passwordKeystring"password"The key for storing the password in the secret
alloy-profiles.remoteConfig.auth.typestring"none"The type of authentication to use for the remote config server
alloy-profiles.remoteConfig.auth.usernamestring""The username to use for the remote config server
alloy-profiles.remoteConfig.auth.usernameFromstring""Raw config for accessing the username
alloy-profiles.remoteConfig.auth.usernameKeystring"username"The key for storing the username in the secret
alloy-profiles.remoteConfig.enabledboolfalseEnable fetching configuration from a remote config server
alloy-profiles.remoteConfig.extraAttributesobject{}Attributes to be added to this collector when requesting configuration
alloy-profiles.remoteConfig.pollFrequencystring"5m"The frequency at which to poll the remote config server for updates
alloy-profiles.remoteConfig.proxyURLstring""The proxy URL to use for the remote config server
alloy-profiles.remoteConfig.secret.createbooltrueWhether to create a secret for the remote config server
alloy-profiles.remoteConfig.secret.embedboolfalseIf true, skip secret creation and embed the credentials directly into the configuration
alloy-profiles.remoteConfig.secret.namestring""The name of the secret to create
alloy-profiles.remoteConfig.secret.namespacestring""The namespace for the secret
alloy-profiles.remoteConfig.urlstring""The URL of the remote config server

Alloy Receiver

KeyTypeDefaultDescription
alloy-receiver.alloy.extraPortslist[]The ports to expose for the Alloy receiver
alloy-receiver.controller.typestring"daemonset"The type of controller to use for the Alloy Receiver instance
alloy-receiver.enabledboolfalseDeploys the Alloy instance for opening receivers to collect application data
alloy-receiver.extraConfigstring""Extra Alloy configuration to be added to the configuration file
alloy-receiver.extraService.enabledboolfalseCreates an extra service for the Alloy receiver. This service will mirror the alloy-receiver service, but its name can be customized to match existing application settings.
alloy-receiver.extraService.fullnamestring""If set, the full name of the extra service to create. This will result in the format <fullname>.
alloy-receiver.extraService.namestring"alloy"The name of the extra service to create. This will result in the format <release-name>-<name>.
alloy-receiver.liveDebugging.enabledboolfalseEnables live debugging for the Alloy instance
alloy-receiver.logging.formatstring"logfmt"Format to use for writing Alloy log lines
alloy-receiver.logging.levelstring"info"Level at which Alloy log lines should be written
alloy-receiver.remoteConfig.auth.passwordstring""The password to use for the remote config server
alloy-receiver.remoteConfig.auth.passwordFromstring""Raw config for accessing the password
alloy-receiver.remoteConfig.auth.passwordKeystring"password"The key for storing the password in the secret
alloy-receiver.remoteConfig.auth.typestring"none"The type of authentication to use for the remote config server
alloy-receiver.remoteConfig.auth.usernamestring""The username to use for the remote config server
alloy-receiver.remoteConfig.auth.usernameFromstring""Raw config for accessing the username
alloy-receiver.remoteConfig.auth.usernameKeystring"username"The key for storing the username in the secret
alloy-receiver.remoteConfig.enabledboolfalseEnables fetching configuration from a remote config server
alloy-receiver.remoteConfig.extraAttributesobject{}Attributes to be added to this collector when requesting configuration
alloy-receiver.remoteConfig.pollFrequencystring"5m"The frequency at which to poll the remote config server for updates

Alloy Singleton

KeyTypeDefaultDescription
alloy-singleton.controller.replicasint1The number of replicas for the Alloy Singleton instance. Must remain a single instance to avoid duplicate data.
alloy-singleton.controller.typestring"deployment"The type of controller to use for the Alloy Singleton instance
alloy-singleton.enabledboolfalseDeploys the Alloy instance for data sources required to be deployed on a single replica
alloy-singleton.extraConfigstring""Extra Alloy configuration to be added to the configuration file
alloy-singleton.liveDebugging.enabledboolfalseEnables live debugging for the Alloy instance
alloy-singleton.logging.formatstring"logfmt"Format to use for writing Alloy log lines
alloy-singleton.logging.levelstring"info"Level at which Alloy log lines should be written
alloy-singleton.remoteConfig.auth.passwordstring""The password to use for the remote config server
alloy-singleton.remoteConfig.auth.passwordFromstring""Raw config for accessing the password
alloy-singleton.remoteConfig.auth.passwordKeystring"password"The key for storing the password in the secret
alloy-singleton.remoteConfig.auth.typestring"none"The type of authentication to use for the remote config server
alloy-singleton.remoteConfig.auth.usernamestring""The username to use for the remote config server
alloy-singleton.remoteConfig.auth.usernameFromstring""Raw config for accessing the username
alloy-singleton.remoteConfig.auth.usernameKeystring"username"The key for storing the username in the secret
alloy-singleton.remoteConfig.enabledboolfalseEnables fetching configuration from a remote config server
alloy-singleton.remoteConfig.extraAttributesobject{}Attributes to be added to this collector when requesting configuration
alloy-singleton.remoteConfig.pollFrequencystring"5m"The frequency at which to poll the remote config server for updates
alloy-singleton.remoteConfig.proxyURLstring""The proxy URL to use for the remote config server
alloy-singleton.remoteConfig.secret.createbooltrueWhether to create a secret for the remote config server
alloy-singleton.remoteConfig.secret.embedboolfalseIf true, skip secret creation and embed the credentials directly into the configuration
alloy-singleton.remoteConfig.secret.namestring""The name of the secret to create
alloy-singleton.remoteConfig.secret.namespacestring""The namespace for the secret
alloy-singleton.remoteConfig.urlstring""The URL of the remote config server

Additional configuration sources

Each collector also has the ability to specify additional configuration sources. These are specified within the Alloy instance’s own section in the values file:

NameAssociated valuesDescription
Extra configurationalloy-___.extraConfigAdditional configuration to be added to the configuration file. Use this for adding custom configuration, but do not use it to modify existing configuration.
Remote configurationalloy-___.remoteConfigConfiguration for fetching remotely defined configuration. To configure, refer to Grafana Fleet Management.
Loggingalloy-___.loggingConfiguration for logging.
Live debuggingalloy-___.liveDebuggingConfiguration for enabling the Alloy Live Debugging feature.