NSQ integration for Grafana Cloud

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. This integration provides Grafana dashboards and alerts based on the metrics exported by the NSQ statsd integration.

Pre-install configuration for the NSQ integration

In order for the integration to work, point nsqd to Grafana agent statsd exporter with -statsd-address, like this:

nsqd -statsd-address=agent.default.svc.cluster.local:9125

You can also enable e2e metrics with extra flags:

nsqd -e2e-processing-latency-percentile=1.0,0.99,0.95 -e2e-processing-latency-window-time=1m0s

More on end-to-end processing latency in nsq.

Post-install configuration for the NSQ integration

In the Grafana agent configuration file, the agent’s statsd_exporter integration must be enabled and configured the following way:

metrics:
  wal_directory: /tmp/wal
  configs:
    - name: integrations
      remote_write:
        - url: http://cortex:9009/api/prom/push
integrations:
  statsd_exporter:
    enabled: true
    metric_relabel_configs:
      - source_labels: [exported_job]
        target_label: job
        replacement: 'integrations/$1'
      - source_labels: [exported_instance]
        target_label: instance
      - regex: (exported_instance|exported_job)
        action: labeldrop
    mapping_config:
      defaults:
        match_type: glob
        glob_disable_ordering: false
        ttl: 1m30s
      mappings:
       - match: "nsq.*.topic.*.channel.*.message_count"
         name: "nsq_topic_channel_message_count"
         match_metric_type: counter
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"
           channel: "$3"

       - match: "nsq.*.topic.*.channel.*.requeue_count"
         name: "nsq_topic_channel_requeue_count"
         match_metric_type: counter
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"
           channel: "$3"

       - match: "nsq.*.topic.*.channel.*.timeout_count"
         name: "nsq_topic_channel_timeout_count"
         match_metric_type: counter
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"
           channel: "$3"

       - match: "nsq.*.topic.*.channel.*.*"
         name: "nsq_topic_channel_${4}"
         match_metric_type: gauge
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"
           channel: "$3"

      #nsq.<nsq_host>_<nsq_port>.topic.<topic_name>.backend_depth [gauge]
       - match: "nsq.*.topic.*.message_count"
         name: "nsq_topic_message_count"
         help: Total number of messages for the topic
         match_metric_type: counter
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"

       - match: "nsq.*.topic.*.message_bytes"
         name: "nsq_topic_message_bytes"
         help: Total number of bytes of all messages
         match_metric_type: counter
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"

       - match: "nsq.*.topic.*.*" #depth, backend_depth and e2e_processing_latency_<percent>
         name: "nsq_topic_${3}"
         match_metric_type: gauge
         labels:
           instance: "$1"
           job: "nsq"
           topic: "$2"
      # mem
      # nsq.<nsq_host>_<nsq_port>.mem.gc_runs
       - match: "nsq.*.mem.gc_runs"
         name: "nsq_mem_gc_runs"
         match_metric_type: counter
         labels:
           instance: "$1"
           job: "nsq"

       - match: "nsq.*.mem.*"
         name: "nsq_mem_${2}"
         match_metric_type: gauge
         labels:
           instance: "$1"
           job: "nsq"