Menu
Grafana Cloud Infrastructure as Code Ansible Manage Agents with Ansible

Manage Agents across Linux hosts with Grafana Ansible

Monitoring Grafana Agents across multiple Linux hosts can be difficult. To make it easier, you can use the Grafana Agent role with the Grafana Ansible collection. This guide shows how to use the grafana_agent Ansible role to deploy and manage Grafana Agents across multiple Linux hosts so you can monitor them using Grafana Cloud.

Before you begin

Before you begin, you should have:

  • A Grafana Cloud account
  • Linux hosts
  • SSH access to the Linux hosts
  • Account permissions sufficient to install and use Grafana Agent on the Linux hosts

Install the Grafana Ansible collection

The Grafana Agent role is available in the Grafana Ansible collection as of the 1.1.0 release.

To install the Grafana Ansible collection, run this command:

ansible-galaxy collection install grafana.grafana:1.1.0

Create an Ansible inventory file

Next, you will set up your hosts and create an inventory file.

  1. Create your hosts and add public SSH keys to them.

    This example uses eight Linux hosts: two Ubuntu hosts, two CentOS hosts, two Fedora hosts, and two Debian hosts.

  2. Create an Ansible inventory file.

    The Ansible inventory, which resides in a file named inventory, looks similar to this:

    146.190.208.216    # hostname = ubuntu-01
    146.190.208.190    # hostname = ubuntu-02
    137.184.155.128    # hostname = centos-01
    146.190.216.129    # hostname = centos-02
    198.199.82.174     # hostname = debian-01
    198.199.77.93      # hostname = debian-02
    143.198.182.156    # hostname = fedora-01
    143.244.174.246    # hostname = fedora-02
    

    Note: If you are copying the above file, remove the comments (#).

  3. Create an ansible.cfg file within the same directory as inventory, with the following values:

    inventory = inventory  # Path to the inventory file
    private_key_file = ~/.ssh/id_rsa   # Path to my private SSH Key
    remote_user=root
    

Install the Linux Node integration for Grafana Cloud

This example uses the Linux Node integration which provides prebuilt Grafana dashboards. Using an integration is optional.

To install the Linux Node integration and access the dashboards:

  1. In your Grafana Cloud instance, click Integrations and Connections (lightning bolt icon), then search for or navigate to the Linux Server tile.

  2. Click the Linux Server tile and click Install Integration.

    The prebuilt dashboards are available.

Configure Grafana Agent

To configure Agent:

  1. Create a file named agent-config.yml within the same directory as ansible.cfg and inventory.

  2. Add the following configuration.

    logs:
      configs:
        - name: default
          clients:
             - basic_auth:
                 password: <Grafana.com API Key>
                 username: <Logs User ID>
               url:  https://<Loki URL>/loki/api/v1/push
          positions:
            filename: /tmp/positions.yaml
          target_config:
            sync_period: 10s
          scrape_configs:
            - job_name: varlogs
              static_configs:
                - targets: [localhost]
                  labels:
                    instance: ${HOSTNAME:-default}
                    job: varlogs
                    __path__: /var/log/*log 
    
    metrics:
    configs:
      - name: integrations
        remote_write:
          - basic_auth:
              password: <Grafana.com API Key>
              username: <metrics User ID>
            url: https://<Prometheus URL>/api/prom/push
    
    global:
      scrape_interval: 60s
    wal_directory: /tmp/grafana-agent-wal
    
    integrations:
     node_exporter:
         enabled: true
         instance: ${HOSTNAME:-default}
     prometheus_remote_write:
       - basic_auth:
           password: <Grafana.com API Key>
           username: <metrics User ID>
         url: https://<Prometheus URL>/api/prom/push
    

    Notice that the label instance has been set to the value ${HOSTNAME:-default}, which is substituted by the value of the HOSTNAME environment variable in the Linux host. To read more about the variable substitution, refer to the Grafana Agent node_exporter_config documentation.

    This example scrapes the systemd journal and log files as described in the Linux integration documentation.

Use the Grafana Agent Ansible role

Next you will create an Ansible playbook that calls the grafana_agent role from the grafana.grafana Ansible collection.

To use the Grafana Agent Ansible role:

  1. Create a file named deploy-agent.yml in the same directory as ansible.cfg and inventory and add the configuration below.

    - name: Install Grafana Agent
      hosts: all
    
      tasks:
       - name: Install Grafana Agent
         ansible.builtin.include_role:
           name: grafana.grafana.grafana_agent
         vars:
           agent_config_local_path: agent-config.yml
           systemd_config: |
             [Unit]
             Description=Grafana Agent
             [Service]
             User=grafana-agent
             Environment=HOSTNAME=%H
             ExecStart={{ agent_binary_location }}/agent-{{ linux_architecture }} -config.expand-env -config.file={{ agent_config_location }}/agent-config.yaml
             Restart=always
             [Install]
             WantedBy=multi-user.target
    

    The playbook calls the grafana_agent role from the grafana.grafana Ansible collection. It is passed two variables:

    • agent_config_local_path: Path where the agent configuration resides on local.
    • systemd_config: Contains the systemd service configuration for Grafana Agent.

    Refer to the Grafana Ansible documentation to understand the other variables you can pass to the grafana_agent role.

  2. To run the playbook, run this command:

    ansible-playbook deploy-agent.yml
    

    Note: You can place the deploy-agent.yml, agent-config, ansible.cfg and inventory files in different directories based on your needs.

Check that logs and metrics are being ingested into Grafana Cloud

Logs and metrics will soon be available in Grafana Cloud. To test this, use the Explore feature. Click the Explore icon (compass icon) in the vertical navigation bar.

Check logs

To check logs:

  1. Use the dropdown menu at the top of the page to select your Loki logs data source.

  2. In the log browser, run the query {instance="centos-01"} where centos-01 is the hostname of one of the Linux hosts.

    If you see log lines (shown in the example below), logs are being received.

    If no log lines appear, logs are not being collected.

Check metrics

To check metrics:

  1. Use the dropdown menu at the top of the page to select your Prometheus data source.

  2. Run the query {instance="centos-01"} where centos-01 is the hostname of one of the Linux hosts.

    If you see a metrics graph and table (shown in the example below), metrics are being received.

    If no metrics appear, metrics are not being collected.

View dashboards

Now that you have logs and metrics in Grafana, you can use dashboards to view them. Here’s an example of one of the prebuilt dashboards included with the Linux integration:

Using the Instance dropdown in the dashboard, you can select from the hostnames where you deployed Grafana Agent and start monitoring them.

Summary

The grafana_agent Ansible role makes it easy to deploy Grafana Agents across various machines at the same time and ultimately makes it easier to manage these deployments. This example showed eight deployments, but it’s possible to use many more. To use more, update the inventory file and re-run the Ansible playbook (which can also be automated).

To learn more about the Grafana Ansible collection, see its GitHub repository or its documentation. To find related tutorials, see the Grafana Ansible documentation.