Gathering logs from a Linux host using the Grafana Agent
This guide will show you how to install the Grafana Agent on a Linux node and use it to push logs to Grafana Cloud. It will also show you how to configure Grafana Cloud to receive those logs using the integrated Loki functionality and search, explore, and view those logs.
NOTE: Some of the information in this page is identical to and sourced from information that appears in Loki Config/Migrating from Promtail.
- Loki is responsible for storing logs and processing queries. Grafana Cloud includes Loki, so you don’t need to perform a Loki installation, you just need configure some settings within Grafana Cloud so that logs are aggregated and stored correctly. This is what enables log storage, which powers both visualization and querying.
- Grafana Agent is the agent that will be deployed on your Linux node that will send log info to Grafana Cloud.
- A Grafana Cloud account.
- A Linux machine with Grafana Agent installed, as shown in the Monitoring a Linux host using the Linux host integration guide.
- Command line (terminal) access to that Linux machine
- Account permissions sufficient to install and use the Grafana Agent on the Linux machine
NOTE: You will need to add the Grafana Agent user as an owner of any log location you intend to collect from. For example, add the
grafana-agentuser to the group
/var/syslog(the group name might be different on your system because it depends on your Linux distro and the log location) like this:
sudo usermod -a -G adm grafana-agent
Configure Grafana Cloud to receive logs
Create a Grafana Cloud API key with the MetricsPublisher role. Save this information as you will need the API key in a later step.
Open Grafana Cloud. In the side menu, click Configuration (gear icon) and select Data Sources.
In the Data Sources tab of the Configuration page, click Add data source.
Select the Loki tile. Keep this open in a browser tab.
In a new browser tab, open Grafana Cloud.
In the new tab, go to Integrations and Connections (lightening bolt icon) and select Loki.
Follow the directions in the UI to create an appropriate API key and configure your system.
Click Finish configuration. You’ll need the information displayed, so keep this tab open.
In the original tab, enter the following information from the second tab: Name, URL, User, the API key.
Check the Basic Auth box.
Configure the agent to send logs
Because your Linux machine is already running the agent, configuring it to send logs along with whatever metrics it is already sending is accomplished by modifying the agent configuration YAML file.
- The agent configuration is stored in
/etc/grafana-agent.yaml. Open the file and add the following code block. It should be placed below the Prometheus (if it exists) and Integrations sections (created when you installed an integration). The new section should start at the root-level of indentation (all the way at the left margin in the file).
<Your Grafana.com API Key>and
<User> with your API key and the user number you found while creating the Loki data source in Grafana Cloud. The URL in our sample is for most US-based customers. Yours may differ. Use the URL you found while configuring your Grafana Cloud in the previous section.
logs: configs: - name: default positions: filename: /tmp/positions.yaml scrape_configs: - job_name: varlogs static_configs: - targets: [localhost] labels: job: varlogs __path__: /var/log/*log clients: - url: https://logs-prod-us-central1.grafana.net/loki/api/v1/push basic_auth: username: <User> password: <Your Grafana.com API Key>
This example will scrape and send info from all logs in
/var/log that end in
log. They are labeled with
varlogs as the job and job_name.
NOTE: Read Labels to learn how to use labels effectively for the best experience.
- Restart the agent for you changes to take effect.
sudo systemctl restart grafana-agent.service
You can add additional sections for logs in other locations or with other filenames. For example, the following section is for
dmesg, which you would place in the
static_configs: section and before the
- job_name: dmesg static_configs: - targets: [localhost] labels: job: dmesg __path__: /var/log/dmesg
In the following example, the job is scraping logs for a minecraft server with logs stored in a subdirectory of the
/home directory of a special minecraft user.
- job_name: minecraftlog static_configs: - targets: [localhost] labels: job: minecraft __path__: /home/MCuser/minecraft/logs/latest.log
Check that logs are being ingested into Grafana Cloud
Within minutes, logs should begin to be available in Grafana Cloud. To test this, use the Explore feature.
- Click the Explore icon (looks like compass points) in the sidebar to start. This takes you to the Explore page.
- At the top of the page, use the dropdown menu to select your Loki logs data source.
The image above used the Log labels dropdown to find the entry for
/var/log/syslog, which is where most of our logs are aggregated.
If no log labels appear, logs are not being collected. If labels are listed, this confirms that logs are being received.
If logs are not displayed after several minutes, ensure the agent is running and check your steps for typos.
Configure a dashboard
Here we will add a dashboard panel for our syslog to an existing Linux Node dashboard that we set up in Monitoring a Linux host using Prometheus and node_exporter. We are only using this dashboard as an example. You can add a logs panel to any dashboard that you can edit; you are not restricted to using this one.
It looks like this.
NOTE: This dashboard was originally imported. As it is, it will be automatically updated to the latest version of this dashboard if we stick with all the defaults. For our example, we are going to make this dashboard editable so that we can add a panel. Note that doing so will prevent the dashboard from receiving future automatic updates.
Make the dashboard editable
To make this dashboard editable, click the settings icon at the top (not in the side panel) of the page (it looks like a gear). In the General settings tab that opens, click Make editable. For our example, we do not need to edit any other settings, so click Save dashboard to continue and in the pop up click Save. Your dashboard is now editable.
If you are not automatically sent back to the dashboard, click the arrow at the top left of the page to return to the dashboard.
Add a panel
Click the Add panel icon at the top of the screen (it looks like a mini graph panel with a + sign on it). A new empty panel appears.
To open the panel settings and configure the new panel, in the new panel, click Add new panel.
You can set a panel title at the right, a description, and you have multiple visualization and display settings available. For simplicity, our new panel will be titled “Syslog”.
To make this new panel display logs, click to to expand the Visualization options on the right and click Logs to select it.
Enter this query into the Log labels box and then click anywhere outside of the box. The log entries should appear in the new panel example in the page.
Here’s what the Edit Panel page should look like now. You can also click on the down arrow next to Log labels to explore the labels being received and create your own panels using other logs and so on.
To finish creating the panel, click Apply at the top of the page.
The panel will be created at the top of the page and may not stretch across the page. Panels are movable and resizable using your cursor to drag and resize as you would a window on your computer’s desktop. Here’s what ours looks like after we resized and put it just below the first row of our dashboard.
Complete instructions for creating a dashboard panel (indeed, multiple panels to create a dashboard) are available in the Grafana Add a panel documentation.
See how to use LogQL to implement alerting and recording rules.
Using these alerts is possible within Grafana Cloud by configuring your alerts as shown above, but from within Grafana Alerting.
Related Grafana Cloud resources
How to set up and visualize synthetic monitoring at scale with Grafana Cloud
Learn how to use Kubernetes, Grafana Loki, and Grafana Cloud’s synthetic monitoring feature to set up your infrastructure's checks in this GrafanaCONline session.
Using Grafana Cloud to drive manufacturing plant efficiency
This GrafanaCONline session tells how Grafana helps a 75-year-old manufacturing company with product quality and equipment maintenance.