Grafana Cloud Quickstart GuidesGathering logs from a Linux host using Promtail

Gathering logs from a Linux host using Promtail

This guide will show you how to install Promtail on a Linux node and use it to push logs to Grafana Cloud. It will also show you how to configure Grafana Cloud to receive those logs using the integrated Loki functionality and search, explore, and view those logs.

NOTE: Some of the information in this page is identical to and sourced from information that appears in Set up Loki and collect logs with Promtail or Grafana Agent.

For clarification:

  • Loki is the main server, responsible for storing logs and processing queries. Grafana Cloud includes Loki, so you don’t need to perform a Loki installation, you just need configure some settings within Grafana Cloud so that logs are aggregated and stored correctly. This is what enables log storage, which powers both visualization and querying.
  • Promtail is the agent, developed by the Loki team and with releases that correspond to Loki releases.

Prerequisites

  • A Grafana Cloud account, as shown in Quickstarts.
  • A Grafana Cloud API key with the MetricsPublisher role
  • A Linux machine
  • Command line (terminal) access to that Linux machine
  • Account permissions sufficient to install and use Docker on the Linux machine

Configure Grafana Cloud with a Loki data source

Open Grafana Cloud. In the side menu, from Settings (looks like a gear) select Data Sources.

On the Configuration page that opens, in the Data Sources tab (which you should already be in), click Add data source.

From the list of options, select Loki. Keep this open in a browser tab.

Open a different browser tab and open Grafana Cloud. In this tab:

  1. In the side menu, from Onboarding (looks like a lightning bolt) select Walkthrough.
  2. Find and select Loki, scroll down and click Next step.
  3. Click Send Logs. In the Loki box, click Details.
  4. Find the listed settings for your organization.

Go back to the previous tab where we were configuring the Loki data source. Enter the information you found for Name, URL, User, the API key you created earlier, and check the Basic Auth box.

Save and move to Install and configure Promtail.

Install and configure Promtail

Before you begin, you must create a configuration file. Our example is a Linux YAML file called config.yaml and saved in /etc/promtail/. Use these contents, with your newly-created API key from Step 1 replacing <Your Grafana.com API Key> and <User> with the user number you found while creating the Loki data source in Grafana Cloud.

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://<User>:<Your Grafana.com API Key>@logs-prod-us-central1.grafana.net/api/prom/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log

NOTE: Read Loki label best practices to learn how to use labels effectively for the best experience.

Loki is open source, and therefore its agent Promtail is as well. You are welcome to open the releases page of the official Loki GitHub repo to find and download the latest binary release of Promtail for your system architecture, but it is much easier to install and run using Docker, like this:

docker run --name promtail --volume "$PWD/promtail:/etc/promtail" --volume "/var/log:/var/log" grafana/promtail:master -config.file=/etc/promtail/config.yaml

Check that logs are being ingested into Grafana Cloud

Within minutes, logs should begin to be available in Grafana Cloud. To test this, use the Explore feature. Click the Explore icon (looks like compass points) in the sidebar to start. This takes you to the Explore page, which looks like this.

The Grafana Explore page

At the top of the page, use the dropdown menu to select your Loki logs data source.

The image above used the Log labels dropdown to find the entry for /var/log/syslog, which is where most of our logs are aggregated.

Note the query we use here, as we will use it again later to create a panel in an existing dashboard:

{filename="/var/log/syslog"}

If no log labels appear, logs are not being collected. If labels are listed, this confirms that logs are being received.

If logs are not displayed after several minutes, check your steps for typos and whether Promtail is running on the Linux machine.

Configure a dashboard

Here we will add a dashboard panel for our syslog to an existing Linux Node dashboard that we set up in Monitoring a Linux host using Prometheus and node_exporter. We are only using this dashboard as an example. You can add a logs panel to any dashboard that you can edit; you are not restricted to using this one.

It looks like this.

Imported dashboard

NOTE: This dashboard was originally imported. As it is, it will be automatically updated to the latest version of this dashboard if we stick with all the defaults. For our example, we are going to make this dashboard editable so that we can add a panel. Note that doing so will prevent the dashboard from receiving future automatic updates.

Make the dashboard editable

To make this dashboard editable, click the settings icon at the top (not in the side panel) of the page (it looks like a gear). In the General settings tab that opens, click Make editable. For our example, we do not need to edit any other settings, so click Save dashboard to continue and in the pop up click Save. Your dashboard is now editable.

If you are not automatically sent back to the dashboard, click the arrow at the top left of the page to return to the dashboard.

Add a panel

Click the Add panel icon at the top of the screen (it looks like a mini graph panel with a + sign on it). A new empty panel appears.

To open the panel settings and configure the new panel, in the new panel, click Add new panel.

You can set a panel title at the right, a description, and you have multiple visualization and display settings available. For simplicity, our new panel will be titled “Syslog”.

To make this new panel display logs, click to to expand the Visualization options on the right and click Logs to select it.

Enter this query into the Log labels box and then click anywhere outside of the box. The log entries should appear in the new panel example in the page.

{filename="/var/log/syslog"}

Here’s what the Edit Panel page should look like now. You can also click on the down arrow next to Log labels to explore the labels being received and create your own panels using other logs and so on.

Edit log panel being added to dashboard

To finish creating the panel, click Apply at the top of the page.

The panel will be created at the top of the page and may not stretch across the page. Panels are movable and resizable using your cursor to drag and resize as you would a window on your computer’s desktop. Here’s what ours looks like after we resized and put it just below the first row of our dashboard.

Finished log panel added to dashboard

Next steps

Complete instructions for creating a dashboard panel (indeed, multiple panels to create a dashboard) are available in the Grafana Add a panel documentation.

Create alerts

See how to use LogQL and the Ruler for Loki alerting.

Using these alerts is possible within Grafana Cloud by configuring your alerts as show above, but from within Grafana Cloud Alerting.

Optional: reduce log info sent to Grafana Cloud

To limit what is sent to Grafana Cloud and potentially reduce spending as a result, you can drop log lines in Promtail before sending them to Loki. This is especially useful for expensive or unnecessary logs or to drop lines that are too long.

To do this, create a section in the Promtail config.yaml that uses a regex in Go syntax matching what you want dropped.

For example, here is the same configuration file used above, but with a new section at the bottom, pipeline_stages: that will drop lines from /var/log/test.log that are longer than 501 characters.

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://<User>:<Your Grafana.com API Key>@logs-prod-us-central1.grafana.net/api/prom/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log
  pipeline_stages:
    - match:
        selector: '{filename="/var/log/test.log"} |~ ".{501}"'
        action: drop

Our example is based on this example, which gives several more examples. Learn more about how to configure Promtail in the configuring Promtail documentation on GitHub.