Collect logs with Grafana Agent
Loki is the main server, responsible for storing logs and processing queries. Grafana Cloud includes Loki, so you don’t need to perform a Loki installation, you just need configure some settings within Grafana Cloud so that logs are aggregated and stored correctly. This is what enables log storage, which powers both visualization and querying.
Promtail is the agent, developed by the Loki team and with releases that correspond to Loki releases. The Grafana Agent includes Promtail capabilities.
Collecting logs into Grafana Cloud is a two-step process. You must:
- Configure Grafana Cloud
- Install and configure a way to send logs; here are the options in this page:
- Configure the Grafana Agent to send logs from a single node
- Configure the Grafana Agent to send logs from a Kubernetes cluster
If you want to use Promtail instead of the Grafana Agent to send logs, see Collect logs with Promtail.
Step 1. Configure Grafana Cloud to receive the logs
To begin, Create a Grafana Cloud API key with the MetricsPublisher role. Save this information as you will need the API key in a later step.
Open Grafana Cloud. In the side menu, from Settings (looks like a gear) select Data Sources.
On the Configuration page that opens, in the Data Sources tab (which you should already be in), click Add data source.
From the list of options, select Loki. Keep this open in a browser tab.
Open a different browser tab and open Grafana Cloud. In this tab:
- In the side menu, from Onboarding (looks like a lightning bolt) select Walkthrough.
- Find and select Loki, scroll down and click Next: Configure service.
- Follow the directions in the UI to create an appropriate API key and configure your system. Click Finish configuration.
Go back to the previous tab where we were configuring the Loki data source. Enter the information you found for Name, URL, User, the API key you created earlier, and check the Basic Auth box.
Save and move to Step 2.
Step 2. Configure the Grafana Agent to send logs
This option covers configuring the Grafana Agent to send logs and assumes you are collecting logs from a Linux host.
There are two options listed here for configuring the Grafana Agent:
- Install and send logs from a single node
- Install and send logs from a Kubernetes cluster
Option A. Install the agent on a single node and send logs
The contents of this section come from the Gathering logs from a Linux host using the Grafana Agent quickstart.
Because your Linux machine is already running the agent, configuring it to send logs along with whatever metrics it is already sending is accomplished by modifying the agent configuration YAML file.
The agent configuration is stored in
/etc/grafana-agent.yaml. Open the file and add this new section, below the Prometheus section (if it exists) and the Integrations section (created when you installed an integration). The new section should start at the root-level of indentation (all the way at the left margin in the file).
Use these contents, with your newly-created API key from Step 1 replacing
<Your Grafana.com API Key>and
<User> with the user number you found while creating the Loki data source in Grafana Cloud. The URL in our sample is for most US-based customers. Yours may differ. Use the URL you found while configuring your Grafana Cloud in the previous section.
loki: configs: - name: default positions: filename: /tmp/positions.yaml scrape_configs: - job_name: varlogs static_configs: - targets: [localhost] labels: job: varlogs __path__: /var/log/*log clients: - url: http://logs-prod-us-central1.grafana.net/loki/api/v1/push basic_auth: username: <User> password: <Your Grafana.com API Key>
This example will scrape and send info from all logs in
/var/log that end in
log. They are labeled with
varlogs as the job and job_name.
NOTE: Read Loki label best practices to learn how to use labels effectively for the best experience.
You can add additional sections for logs in other locations or with other filenames. For example, here’s one for
dmesg, which you would place in the
static_configs: section and before the
- job_name: dmesg static_configs: - targets: [localhost] labels: job: dmesg __path__: /var/log/dmesg
Here is another example, scraping logs for a minecraft server with logs stored in a subdirectory of the
/home directory of a special minecraft user.
- job_name: minecraftlog static_configs: - targets: [localhost] labels: job: minecraft __path__: /home/MCuser/minecraft/logs/latest.log
Anytime you change the agent configuration, you must restart the agent for the new configuration to take effect.
sudo systemctl restart grafana-agent.service
Option B. Install the agent and send logs from a Kubernetes cluster
To learn how to roll out the Grafana Agent into a Kubernetes cluster to ship logs, please see the Grafana Agent Logs Kubernetes Quickstart
Querying logs and creating panels
Once you have Promtail up and running on your log source, give it some time to start collecting logs. Eventually, you will be able to query logs and create panels inside dashboards using Loki as a datasource.
Related Grafana Cloud resources
How to set up and visualize synthetic monitoring at scale with Grafana Cloud
Learn how to use Kubernetes, Grafana Loki, and Grafana Cloud’s synthetic monitoring feature to set up your infrastructure's checks in this GrafanaCONline session.
Using Grafana Cloud to drive manufacturing plant efficiency
This GrafanaCONline session tells how Grafana helps a 75-year-old manufacturing company with product quality and equipment maintenance.