Data configurationGrafana integrationsIntegrations referenceGrafana Agent integration

Grafana Agent integration for Grafana Cloud

The Grafana agent integration lets you monitor metrics and logs of a Grafana agent instance

Install Grafana Agent integration for Grafana Cloud

  1. In your Grafana instance, Click Integrations and Connections (lightning bolt icon), then click on install integration on Grafana Agent tile.
  2. Wait for the integration to be installed and then follow the steps shown to setup Grafana Agent to automatically scrape and send Grafana Agent metrics to your Cloud Instance.

Post-install configuration for the Grafana Agent integration

This integration supports metrics and logs for Docker containers. In order to see those signals correlated on the same dashboard, job and instance labels must match for agent(metrics) and logs in the Agent configuration file. The default instance value in the agent integration is the hostname of the machine the agent is running on.

The grafana agent integration needs to be enabled in the grafana agent config file. In order to enable logs - the journal scraping needs to be enabled in case of linux and windows event log scraping needs to enabled for windows. Config snippet for both cases are shown below.

Linux

integrations:
  agent:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels:
          - agent_hostname
        target_label: instance
  prometheus_remote_write:
    - url: http://cortex:9009/api/prom/push

logs:
  configs:
    - name: agent
      clients:
        - url: http://loki:3100/loki/api/v1/push
      positions:
        filename: /tmp/positions.yaml
      target_config:
        sync_period: 10s
      scrape_configs:
      - job_name: integrations/agent
        journal:
          max_age: 24h
          labels:
            instance: <hostname>
            job: integrations/agent
        pipeline_stages:
          - match:
              selector: '{unit!="grafana-agent.service"}'
              action: drop
              drop_counter_reason: only_keep_grafana_agent_logs
          - regex:
              expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
          - labels:
              level: log_level
        relabel_configs:
          - source_labels: ['__journal__systemd_unit']
            target_label: 'unit'

Windows

integrations:
  agent:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels:
          - agent_hostname
        target_label: instance
  prometheus_remote_write:
    - url: http://cortex:9009/api/prom/push

logs:
  configs:
    - name: agent
      clients:
        - url: http://shb-mac:3100/api/prom/push
      positions:
        filename: /tmp/positions.yaml
      target_config:
        sync_period: 10s
      scrape_configs:
      - job_name: integrations/agent
        windows_events:
          use_incoming_timestamp: true
          bookmark_path: "./bookmark-application.xml"
          eventlog_name: "Application"
          xpath_query: "*[System[Provider[@Name='Grafana Agent']]]"
          labels:
            job: integrations/agent
        relabel_configs:
          - source_labels: ['computer']
            target_label: 'instance'
          - replacement: grafana-agent.service
            target_label: unit
        pipeline_stages:
        - json:
            expressions:
              message: message
        - regex:
            expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
        - labels:
            level: log_level
        - output:
            source: message

Dashboards

This integration includes the following dashboards.

  • Grafana Agent Logs
  • Grafana Agent Operational
  • Grafana Agent Overview
  • Grafana Agent Remote Write
  • Grafana Agent Tracing Pipeline

Grafana Agent Overview

image

Grafana Agent Logs

image

Grafana Agent Operational

image

Grafana Agent Prometheus Remote Write

image

Grafana Agent Tracing Pipeline

image

Metrics

Below is a list of the metrics used on the dashboards and in alerts of the integration and are automatically written to your Grafana Cloud instance.

  • agent_build_info
  • agent_tcp_connections
  • agent_wal_samples_appended_total
  • agent_wal_storage_active_series
  • go_gc_duration_seconds_count
  • go_goroutines
  • go_memstats_heap_inuse_bytes
  • process_cpu_seconds_total
  • process_start_time_seconds
  • prometheus_remote_storage_enqueue_retries_total
  • prometheus_remote_storage_highest_timestamp_in_seconds
  • prometheus_remote_storage_queue_highest_sent_timestamp_seconds
  • prometheus_remote_storage_samples_dropped_total
  • prometheus_remote_storage_samples_failed_total
  • prometheus_remote_storage_samples_pending
  • prometheus_remote_storage_samples_retried_total
  • prometheus_remote_storage_samples_total
  • prometheus_remote_storage_sent_batch_duration_seconds_bucket
  • prometheus_remote_storage_sent_batch_duration_seconds_count
  • prometheus_remote_storage_sent_batch_duration_seconds_sum
  • prometheus_remote_storage_shard_capacity
  • prometheus_remote_storage_shards
  • prometheus_remote_storage_shards_desired
  • prometheus_remote_storage_shards_max
  • prometheus_remote_storage_shards_min
  • prometheus_remote_storage_succeeded_samples_total
  • prometheus_sd_discovered_targets
  • prometheus_target_interval_length_seconds_count
  • prometheus_target_interval_length_seconds_sum
  • prometheus_target_scrapes_exceeded_sample_limit_total
  • prometheus_target_scrapes_sample_duplicate_timestamp_total
  • prometheus_target_scrapes_sample_out_of_bounds_total
  • prometheus_target_scrapes_sample_out_of_order_total
  • prometheus_target_sync_length_seconds_sum
  • prometheus_wal_watcher_current_segment
  • traces_exporter_send_failed_spans
  • traces_exporter_sent_spans
  • traces_loadbalancer_backend_outcome
  • traces_loadbalancer_num_backends
  • traces_receiver_accepted_spans
  • traces_receiver_refused_spans

Cost

By connecting your Grafana Agent instance to Grafana Cloud you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.