Grafana Agent integration for Grafana Cloud

The Grafana agent integration lets you monitor metrics and logs of a Grafana agent instance

Install Grafana Agent integration for Grafana Cloud

In your Grafana Cloud instance, click on Connections from the Home menu located near the top left-hand side of the page.
Navigate to the Grafana Agent tile and click on it to review the prerequisites.
When you are ready, follow the steps shown to setup Grafana Agent to automatically scrape and send Grafana Agent metrics to your Cloud Instance.

Post-install configuration for the Grafana Agent integration

This integration supports metrics and logs for Docker containers. In order to see those signals correlated on the same dashboard, job and instance labels must match for agent(metrics) and logs in the Agent configuration file. The default instance value in the agent integration is the hostname of the machine the agent is running on.

The grafana agent integration needs to be enabled in the grafana agent config file. In order to enable logs - the journal scraping needs to be enabled in case of linux and windows event log scraping needs to enabled for windows. Config snippet for both cases are shown below.

Linux

integrations:
  agent:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels:
          - agent_hostname
        target_label: instance
  prometheus_remote_write:
    - url: http://cortex:9009/api/prom/push

logs:
  configs:
    - name: agent
      clients:
        - url: http://loki:3100/loki/api/v1/push
      positions:
        filename: /tmp/positions.yaml
      target_config:
        sync_period: 10s
      scrape_configs:
        - job_name: integrations/agent
          journal:
            max_age: 24h
            labels:
              instance: <hostname>
              job: integrations/agent
          pipeline_stages:
            - match:
                selector: '{unit!="grafana-agent.service"}'
                action: drop
                drop_counter_reason: only_keep_grafana_agent_logs
            - regex:
                expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
            - labels:
                level: log_level
          relabel_configs:
            - source_labels: ['__journal__systemd_unit']
              target_label: 'unit'

Windows

integrations:
  agent:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels:
          - agent_hostname
        target_label: instance
  prometheus_remote_write:
    - url: http://cortex:9009/api/prom/push

logs:
  configs:
    - name: agent
      clients:
        - url: http://shb-mac:3100/api/prom/push
      positions:
        filename: /tmp/positions.yaml
      target_config:
        sync_period: 10s
      scrape_configs:
        - job_name: integrations/agent
          windows_events:
            use_incoming_timestamp: true
            bookmark_path: './bookmark-application.xml'
            eventlog_name: 'Application'
            xpath_query: "*[System[Provider[@Name='Grafana Agent']]]"
            labels:
              job: integrations/agent
          relabel_configs:
            - source_labels: ['computer']
              target_label: 'instance'
            - replacement: grafana-agent.service
              target_label: unit
          pipeline_stages:
            - json:
                expressions:
                  message: message
            - regex:
                expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
            - labels:
                level: log_level
            - output:
                source: message

Dashboards

This integration includes the following dashboards.

Grafana Agent Logs
Grafana Agent Operational
Grafana Agent Overview
Grafana Agent Remote Write
Grafana Agent Tracing Pipeline

Grafana Agent Overview

Grafana Agent Logs

Grafana Agent Operational

Grafana Agent Prometheus Remote Write

Grafana Agent Tracing Pipeline

Metrics

Below is a list of the metrics used on the dashboards and in alerts of the integration and are automatically written to your Grafana Cloud instance.

agent_build_info
agent_tcp_connections
agent_wal_samples_appended_total
agent_wal_storage_active_series
go_gc_duration_seconds_count
go_goroutines
go_memstats_heap_inuse_bytes
process_cpu_seconds_total
process_start_time_seconds
prometheus_remote_storage_enqueue_retries_total
prometheus_remote_storage_highest_timestamp_in_seconds
prometheus_remote_storage_queue_highest_sent_timestamp_seconds
prometheus_remote_storage_samples_dropped_total
prometheus_remote_storage_samples_failed_total
prometheus_remote_storage_samples_pending
prometheus_remote_storage_samples_retried_total
prometheus_remote_storage_samples_total
prometheus_remote_storage_sent_batch_duration_seconds_bucket
prometheus_remote_storage_sent_batch_duration_seconds_count
prometheus_remote_storage_sent_batch_duration_seconds_sum
prometheus_remote_storage_shard_capacity
prometheus_remote_storage_shards
prometheus_remote_storage_shards_desired
prometheus_remote_storage_shards_max
prometheus_remote_storage_shards_min
prometheus_remote_storage_succeeded_samples_total
prometheus_sd_discovered_targets
prometheus_target_interval_length_seconds_count
prometheus_target_interval_length_seconds_sum
prometheus_target_scrapes_exceeded_sample_limit_total
prometheus_target_scrapes_sample_duplicate_timestamp_total
prometheus_target_scrapes_sample_out_of_bounds_total
prometheus_target_scrapes_sample_out_of_order_total
prometheus_target_sync_length_seconds_sum
prometheus_wal_watcher_current_segment
traces_exporter_send_failed_spans
traces_exporter_sent_spans
traces_loadbalancer_backend_outcome
traces_loadbalancer_num_backends
traces_receiver_accepted_spans
traces_receiver_refused_spans

Cost

By connecting your Grafana Agent instance to Grafana Cloud you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.