Collect logs with Promtail
The Grafana Cloud stack includes a logging service powered by Grafana Loki, a Prometheus-inspired log aggregation system. This means that you are not required to run your own Loki environment, though you can ship logs to Grafana Cloud using Promtail or another supported client if you maintain a self-hosted Loki environment. If you want to use the Grafana Agent instead of Promtail to send logs, see Collect logs with Grafana Agent.
Prerequisites
- A Grafana Cloud account
- A Loki environment
- An application or system generating logs
- Docker (optional)
Step 1. Install Promtail
Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.
It primarily:
- Discovers targets
- Attaches labels to log streams
- Pushes them to the Loki instance.
Currently, Promtail can tail logs from two sources: local log files and the systemd journal (on AMD64 machines only).
For the options available to install Promtail, see Install Promtail.
Step 2. Configure Promtail to send logs
There are two options listed in this guide for configuring Promtail:
- Send logs from a standalone host
- Send logs from a Kubernetes cluster
Option A. Send logs from a standalone host
To create the Promtail configuration file, log in to your managed Grafana Cloud environment. Next, select the lightning bolt icon from the left hand menu for Integrations and Connections.
On the main Integrations and Connections page click the “Filter by category” dropdown box located near the top of the page, just below the search field. From those options, select Data Forwarding and then Custom logs.
Next, select the option for a standalone host and name the API key this integration will use:
This will generate the details needed to create the configuration file. The example below is a Linux YAML file called config.yaml
and saved in /etc/promtail/
.
server:
http_listen_port: 0
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
client:
url: https://$InstanceID:$GrafanaCloudAPIKey@$logs-endpoint.grafana.net/api/prom/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*.log
Option B. Send logs from a Kubernetes cluster
To create the Promtail configuration file, log in to your managed Grafana Cloud environment. Next, select the lightning bolt icon from the left hand menu for Integrations and Connections.
On the main Integrations and Connections page click the “Filter by category” dropdown box located near the top of the page, just below the search field. From those options, select Data Forwarding and then Custom logs.
Next, select the option for a Kubernetes cluster and name the API key this integration will use:
This will generate the details needed to create the configuration file, using the following format:
curl -fsS https://raw.githubusercontent.com/grafana/loki/main/tools/promtail.sh | sh -s <$InstanceID> <$Your Grafana Cloud API Key> $Your logs-endpoint.grafana.net default | kubectl apply --namespace=default -f -
Step 3. Confirm logs are being ingested into Grafana Cloud
Within several minutes, logs should begin to be available in Grafana Cloud. To test this, use the Explore feature.
Click the compass Explore icon from the left sidebar menu to start. This takes you to the Explore page.
At the top of the page, use the dropdown menu to select your Loki logs data source. This should be named grafanacloud-$yourstackname-logs
.
The image below shows the Log browser dropdown to find the labels for logs being ingested to your Grafana Cloud environment.
If no log labels appear, logs are not being collected. If labels are listed, this confirms that logs are being received.
If logs are not displayed after several minutes, ensure Promtail is running and check your steps for typos. See the Troubleshooting Promtail guide for troubleshooting tips.
In addition to the Log browser dropdown, the Explore user interface also supports autocomplete options:
Below is another example of other operators and parsers available. For more details about querying log data, see LogQL: Log query language
Querying logs and creating panels
Once you have Grafana Agent up and running on your log source, give it some time to start collecting logs. Eventually, you will be able to query logs and create panels inside dashboards using Loki as a datasource.
Querying logs is done using LogQL which can be used in both Explore and when creating dashboard panels.
For examples and feature showcases, check out play.grafana.org for ideas and inspiration.
Related Grafana Cloud resources
Intro to Prometheus and Grafana Cloud
Prometheus is taking over the monitoring world! In this webinar, we will start with a quick introduction to the open source project that’s the de facto standard for monitoring modern, cloud native systems.
How to set up and visualize synthetic monitoring at scale with Grafana Cloud
Learn how to use Kubernetes, Grafana Loki, and Grafana Cloud’s synthetic monitoring feature to set up your infrastructure's checks in this GrafanaCONline session.
Using Grafana Cloud to drive manufacturing plant efficiency
This GrafanaCONline session tells how Grafana helps a 75-year-old manufacturing company with product quality and equipment maintenance.