Grafana Cloud quickstart guidesGathering logs from a Linux host using the Grafana Agent

Gathering logs from a Linux host using the Grafana Agent

This guide will show you how to install the Grafana Agent on a Linux node and use it to push logs to Grafana Cloud. It will also show you how to configure Grafana Cloud to receive those logs using the integrated Loki functionality and search, explore, and view those logs.

NOTE: Some of the information in this page is identical to and sourced from information that appears in Loki Config/Migrating from Promtail.

For clarification:

  • Loki is the main server, responsible for storing logs and processing queries. Grafana Cloud includes Loki, so you don’t need to perform a Loki installation, you just need configure some settings within Grafana Cloud so that logs are aggregated and stored correctly. This is what enables log storage, which powers both visualization and querying.
  • Grafana Agent is the agent that will be deployed on your Linux node that will send log info to Grafana Cloud.

Prerequisites

NOTE: You will need to add the Grafana Agent user as an owner of any log location you intend to collect from. For example, add the grafana-agent user to the group adm which owns /var/syslog (the group name might be different on your system because it depends on your Linux distro and the log location) like this:

sudo usermod -a -G adm grafana-agent

Configure Grafana Cloud to receive logs

To begin, Create a Grafana Cloud API key with the MetricsPublisher role. Save this information as you will need the API key in a later step.

Open Grafana Cloud. In the side menu, from Settings (looks like a gear) select Data Sources.

On the Configuration page that opens, in the Data Sources tab (which you should already be in), click Add data source.

From the list of options, select Loki. Keep this open in a browser tab.

Open a different browser tab and open Grafana Cloud. In this tab:

  1. In the side menu, from Onboarding (looks like a lightning bolt) select Walkthrough.
  2. Find and select Loki, scroll down and click Next: Configure service.
  3. Follow the directions in the UI to create an appropriate API key and configure your system. Click Finish configuration.

Go back to the previous tab where we were configuring the Loki data source. Enter the information you found for Name, URL, User, the API key you created earlier, and check the Basic Auth box.

Configure the agent to send logs

Because your Linux machine is already running the agent, configuring it to send logs along with whatever metrics it is already sending is accomplished by modifying the agent configuration YAML file.

The agent configuration is stored in /etc/grafana-agent.yaml. Open the file and add this new section, below the Prometheus section (if it exists) and the Integrations section (created when you installed an integration). The new section should start at the root-level of indentation (all the way at the left margin in the file). Use these contents, with your newly-created API key replacing <Your Grafana.com API Key>and <User> with the user number you found while creating the Loki data source in Grafana Cloud. The URL in our sample is for most US-based customers. Yours may differ. Use the URL you found while configuring your Grafana Cloud in the previous section.

loki:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: varlogs
        static_configs:
          - targets: [localhost]
            labels:
              job: varlogs
              __path__: /var/log/*log
    clients:
      - url: http://logs-prod-us-central1.grafana.net/loki/api/v1/push
        basic_auth:
          username: <User>
          password: <Your Grafana.com API Key>

This example will scrape and send info from all logs in /var/log that end in log. They are labeled with varlogs as the job and job_name.

NOTE: Read Loki label best practices to learn how to use labels effectively for the best experience.

You can add additional sections for logs in other locations or with other filenames. For example, here’s one for dmesg, which you would place in the static_configs: section and before the clients: section.

      - job_name: dmesg
        static_configs:
          - targets: [localhost]
            labels:
              job: dmesg
              __path__: /var/log/dmesg

Here is another example, scraping logs for a minecraft server with logs stored in a subdirectory of the /home directory of a special minecraft user.

      - job_name: minecraftlog
        static_configs:
          - targets: [localhost]
            labels:
              job: minecraft
              __path__: /home/MCuser/minecraft/logs/latest.log

Anytime you change the agent configuration, you must restart the agent for the new configuration to take effect.

sudo systemctl restart grafana-agent.service

Check that logs are being ingested into Grafana Cloud

Within minutes, logs should begin to be available in Grafana Cloud. To test this, use the Explore feature. Click the Explore icon (looks like compass points) in the sidebar to start. This takes you to the Explore page, which looks like this.

The Grafana Explore page

At the top of the page, use the dropdown menu to select your Loki logs data source.

The image above used the Log labels dropdown to find the entry for /var/log/syslog, which is where most of our logs are aggregated.

Note the query we use here, as we will use it again later to create a panel in an existing dashboard:

{filename="/var/log/syslog"}

If no log labels appear, logs are not being collected. If labels are listed, this confirms that logs are being received.

If logs are not displayed after several minutes, check your steps for typos and whether the agent is running on the Linux machine.

Configure a dashboard

Here we will add a dashboard panel for our syslog to an existing Linux Node dashboard that we set up in Monitoring a Linux host using Prometheus and node_exporter. We are only using this dashboard as an example. You can add a logs panel to any dashboard that you can edit; you are not restricted to using this one.

It looks like this.

Imported dashboard

NOTE: This dashboard was originally imported. As it is, it will be automatically updated to the latest version of this dashboard if we stick with all the defaults. For our example, we are going to make this dashboard editable so that we can add a panel. Note that doing so will prevent the dashboard from receiving future automatic updates.

Make the dashboard editable

To make this dashboard editable, click the settings icon at the top (not in the side panel) of the page (it looks like a gear). In the General settings tab that opens, click Make editable. For our example, we do not need to edit any other settings, so click Save dashboard to continue and in the pop up click Save. Your dashboard is now editable.

If you are not automatically sent back to the dashboard, click the arrow at the top left of the page to return to the dashboard.

Add a panel

Click the Add panel icon at the top of the screen (it looks like a mini graph panel with a + sign on it). A new empty panel appears.

To open the panel settings and configure the new panel, in the new panel, click Add new panel.

You can set a panel title at the right, a description, and you have multiple visualization and display settings available. For simplicity, our new panel will be titled “Syslog”.

To make this new panel display logs, click to to expand the Visualization options on the right and click Logs to select it.

Enter this query into the Log labels box and then click anywhere outside of the box. The log entries should appear in the new panel example in the page.

{filename="/var/log/syslog"}

Here’s what the Edit Panel page should look like now. You can also click on the down arrow next to Log labels to explore the labels being received and create your own panels using other logs and so on.

Edit log panel being added to dashboard

To finish creating the panel, click Apply at the top of the page.

The panel will be created at the top of the page and may not stretch across the page. Panels are movable and resizable using your cursor to drag and resize as you would a window on your computer’s desktop. Here’s what ours looks like after we resized and put it just below the first row of our dashboard.

Finished log panel added to dashboard

Next steps

Complete instructions for creating a dashboard panel (indeed, multiple panels to create a dashboard) are available in the Grafana Add a panel documentation.

Create alerts

See how to use LogQL and the Ruler for Loki alerting.

Using these alerts is possible within Grafana Cloud by configuring your alerts as show above, but from within Grafana Cloud Alerting.