Data configurationGrafana integrationsIntegrations referenceLinux Node integration

Linux Node integration for Grafana Cloud

The Linux Node integration for Grafana Cloud enables you to collect metrics related to the operating system running on a node, including aspects like CPU usage, load average, memory usage, and disk and networking I/O. It also allows you to use the agent to scrape logs with promtail. This integration includes useful alerts and two prebuilt dashboards to help you monitor and visualize these metrics and logs.

Before you begin

Supported files:

  • syslog
  • kern.log
  • journal logs

You need to add the Grafana agent user as an owner of any log location you intend to collect from. For example, add the grafana-agent user to the group adm which owns /var/syslog with the following command:

sudo usermod -a -G adm grafana-agent

Note: Group name depends on your Linux distro and the log location, please verify your systems group name before adding Grafana Agent as an owner.

Install Linux Node integration for Grafana Cloud

  1. In your Grafana Cloud instance, click Integrations and Connections (lightning bolt icon), then search for or navigate to the Linux Server tile.
  2. Click the Linux Server tile and click Install Integration.
  3. Once the integration is installed, follow the steps on the Configuration Details page to setup Grafana Agent to automatically scrape and send Linux Server metrics to your Grafana Cloud Instance.

Configure scrape jobs (or similar metrics-gathering term)

To see logs and metrics signals correlated on the same dashboard, you must verify that job and instance labels match in node_exporter integration configuration under metrics, and under logs. The job label should be set to integrations/node_exporter. The agent sets this by default for metrics, so you only need to specify this for the log scrapes.

There are three configuration options for the agent to scrape your logs. To view both systemd and OS (/var/log files) logs, use the Scrape systemd and OS logs configuration.

Scrape the systemd journal

This configuration provides a set of predefined labels that are used on the pre-built Linux Server dashboards. This allows you to filter a specific systemd application log using the Systemd Unit dropdown filter.

integrations:
  node_exporter:
    enabled: true
    relabel_configs:
    - replacement: hostname
      source_labels:
      - __address__
      target_label: instance

logs:
  configs:
  - name: integrations
    scrape_configs:
    - job_name: integrations/node_exporter_journal_scrape
      journal:
        max_age: 24h
        labels:
          instance: hostname
          job: integrations/node_exporter
      relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__boot_id']
        target_label: 'boot_id'
      - source_labels: ['__journal__transport']
        target_label: 'transport'

Scrape log files directly

This configuration allows you to scrape log files directly. Use this configuration if the systemd journal configuration is not an option.

integrations:
  node_exporter:
    enabled: true
    relabel_configs:
    - replacement: hostname
      source_labels:
      - __address__
      target_label: instance

logs:
  configs:
  - name: integrations
    scrape_configs:
    - job_name: integrations/node_exporter_direct_scrape
      static_configs:
      - targets:
        - localhost
        labels:
          instance: hostname
          __path__: /var/log/{syslog,messages,auth.log,secure,kern.log}
          job: integrations/node_exporter

This configuration allows you to view both systemd and OS (/var/log files) logs. Use this configuration to scrape the systemd journal and log files directly.

integrations:
  node_exporter:
    enabled: true
    relabel_configs:
    - replacement: hostname
      source_labels:
      - __address__
      target_label: instance

logs:
  configs:
  - name: integrations
    scrape_configs:
    - job_name: integrations/node_exporter_direct_scrape
      static_configs:
      - targets:
        - localhost
        labels:
          instance: hostname
          __path__: /var/log/{syslog,messages,*.log}
          job: integrations/node_exporter
    - job_name: integrations/node_exporter_journal_scrape
      journal:
        max_age: 24h
        labels:
          instance: hostname
          job: integrations/node_exporter
      relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__boot_id']
        target_label: 'boot_id'
      - source_labels: ['__journal__transport']
        target_label: 'transport'

Dashboards

After you have successfully configured the Linux Node integration, two prebuilt dashboards are installed in your Grafana Cloud instance to help you monitor your Linux Node metrics and logs.

Node dashboard

This dashboard provides information about a specific node including CPU usage, load average, memory usage, and disk space usage.

Linux Node Dashboard

USE Method dashboard

This dashboard is developed based on the USE method, focusing on resources utilization, saturation and error. It provides statistics of the logs collected by Grafana Agent to help you analyze the performance of your operating system.

Linux USE method dashboard

Configure alerts

This integrations ships 16 alerts, between warning and critical levels:

AlertDescription
NodeFilesystemAlmostOutOfSpaceWarning: Filesystem has less than 5% space left
NodeFilesystemAlmostOutOfSpaceCritical: Filesystem has less than 3% space left
NodeFilesystemFilesFillingUpWarning: Filesystem is predicted to run out of inodes within the next 24 hours
NodeFilesystemFilesFillingUpCritical: Filesystem is predicted to run out of inodes within the next 4 hours
NodeFilesystemAlmostOutOfFilesWarning: Filesystem has less than 5% inodes left.
NodeFilesystemAlmostOutOfFilesCritical: Filesystem has less than 3% inodes left.
NodeNetworkReceiveErrsWarning: Network interface is reporting many receive errors
NodeNetworkTransmitErrsWarning: Network interface is reporting many transmit errors
NodeHighNumberConntrackEntriesUsedWarning: Number of conntrack are getting close to the limit
NodeTextFileCollectorScrapeErrorWarning: Node Exporter text file collector failed to scrape
NodeClockSkewDetectedWarning: Clock skew detected
NodeClockNotSynchronisingWarning: Clock not synchronizing
NodeRAIDDegradedCritical: RAID Array is degraded
NodeRAIDDiskFailureWarning: Failed device in RAID array
NodeFileDescriptorLimitWarning: Kernel is predicted to exhaust file descriptors limit soon
NodeFileDescriptorLimitCritical: Kernel is predicted to exhaust file descriptors limit soon

Services and metrics captured by Linux Node integration for Grafana Cloud

To see the list of metrics scraped by this integration, refer to the Prometheus metics file.

Cost

By connecting your Linux Server to Grafana Cloud you might incur charges. For more information, use the following links:

For an increase in the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.