Gathering logs from a Linux host using the Grafana Agent
This guide will show you how to install the Grafana Agent on a Linux node and use it to push logs to Grafana Cloud. It will also show you how to configure Grafana Cloud to receive those logs using the integrated Loki functionality and search, explore, and view those logs.
NOTE: Some of the information in this page is identical to and sourced from information that appears in Loki Config/Migrating from Promtail.
- Loki is the main server, responsible for storing logs and processing queries. Grafana Cloud includes Loki, so you don’t need to perform a Loki installation, you just need configure some settings within Grafana Cloud so that logs are aggregated and stored correctly. This is what enables log storage, which powers both visualization and querying.
- Grafana Agent is the agent that will be deployed on your Linux node that will send log info to Grafana Cloud.
- A Grafana Cloud account, as shown in Quickstarts.
- A Grafana Cloud API key with the MetricsPublisher role
- A Linux machine with the Grafana Agent installed, as shown in the Monitoring a Linux host using the Linux host integration quickstart
- Command line (terminal) access to that Linux machine
- Account permissions sufficient to install and use the Grafana Agent on the Linux machine
Configure the agent to send logs
Because your Linux machine is already running the agent, configuring it to send logs along with whatever metrics it is already sending is accomplished by modifying the agent configuration YAML file.
The agent configuration is stored in
/etc/grafana-agent.yaml. Open the file and add this new section, below the Prometheus section (if it exists) and the Integrations section (created when you installed an integration). The new section should start at the root-level of indentation (all the way at the left margin in the file).
loki: configs: - name: default positions: filename: /tmp/positions.yaml scrape_configs: - job_name: varlogs static_configs: - targets: [localhost] labels: job: varlogs __path__: /var/log/*log clients: - url: http://localhost:3100/loki/api/v1/push
This example will scrape and send info from all logs in
/var/log that end in
log. They are labeled with
varlogs as the job and job_name.
NOTE: Read Loki label best practices to learn how to use labels effectively for the best experience.
You can add additional sections for logs in other locations or with other filenames. For example, here’s one for
dmesg, which you would place in the
static_configs: section and before the
- job_name: dmesg static_configs: - targets: [localhost] labels: job: dmesg __path__: /var/log/dmesg
Here is another example, scraping logs for a minecraft server with logs stored in a subdirectory of the
/home directory of a special minecraft user.
- job_name: minecraftlog static_configs: - targets: [localhost] labels: job: minecraft __path__: /home/MCuser/minecraft/logs/latest.log
Anytime you change the agent configuration, you must restart the agent for the new configuration to take effect.
sudo systemctl restart grafana-agent.service
Check that logs are being ingested into Grafana Cloud
Within minutes, logs should begin to be available in Grafana Cloud. To test this, use the Explore feature. Click the Explore icon (looks like compass points) in the sidebar to start. This takes you to the Explore page, which looks like this.
At the top of the page, use the dropdown menu to select your Loki logs data source.
The image above used the Log labels dropdown to find the entry for
/var/log/syslog, which is where most of our logs are aggregated.
Note the query we use here, as we will use it again later to create a panel in an existing dashboard:
If no log labels appear, logs are not being collected. If labels are listed, this confirms that logs are being received.
If logs are not displayed after several minutes, check your steps for typos and whether the agent is running on the Linux machine.
Configure a dashboard
Here we will add a dashboard panel for our syslog to an existing Linux Node dashboard that we set up in Monitoring a Linux host using Prometheus and node_exporter. We are only using this dashboard as an example. You can add a logs panel to any dashboard that you can edit; you are not restricted to using this one.
It looks like this.
NOTE: This dashboard was originally imported. As it is, it will be automatically updated to the latest version of this dashboard if we stick with all the defaults. For our example, we are going to make this dashboard editable so that we can add a panel. Note that doing so will prevent the dashboard from receiving future automatic updates.
Make the dashboard editable
To make this dashboard editable, click the settings icon at the top (not in the side panel) of the page (it looks like a gear). In the General settings tab that opens, click Make editable. For our example, we do not need to edit any other settings, so click Save dashboard to continue and in the pop up click Save. Your dashboard is now editable.
If you are not automatically sent back to the dashboard, click the arrow at the top left of the page to return to the dashboard.
Add a panel
Click the Add panel icon at the top of the screen (it looks like a mini graph panel with a + sign on it). A new empty panel appears.
To open the panel settings and configure the new panel, in the new panel, click Add new panel.
You can set a panel title at the right, a description, and you have multiple visualization and display settings available. For simplicity, our new panel will be titled “Syslog”.
To make this new panel display logs, click to to expand the Visualization options on the right and click Logs to select it.
Enter this query into the Log labels box and then click anywhere outside of the box. The log entries should appear in the new panel example in the page.
Here’s what the Edit Panel page should look like now. You can also click on the down arrow next to Log labels to explore the labels being received and create your own panels using other logs and so on.
To finish creating the panel, click Apply at the top of the page.
The panel will be created at the top of the page and may not stretch across the page. Panels are movable and resizable using your cursor to drag and resize as you would a window on your computer’s desktop. Here’s what ours looks like after we resized and put it just below the first row of our dashboard.
Complete instructions for creating a dashboard panel (indeed, multiple panels to create a dashboard) are available in the Grafana Add a panel documentation.
See how to use LogQL and the Ruler for Loki alerting.
Using these alerts is possible within Grafana Cloud by configuring your alerts as show above, but from within Grafana Cloud Alerting.