Sending logs from the cloud on Grafana Labs

Run the Promtail client on AWS EC2

Sat, 11 Apr 2026 00:53:11 +0000

Run the Promtail client on AWS EC2

In this tutorial we’re going to setup Promtail on an AWS EC2 instance and configure it to sends all its logs to a Grafana Loki instance.

Caution
Promtail has been deprecated and is in Long-Term Support (LTS) through February 28, 2026. Promtail will reach an End-of-Life (EOL) on March 2, 2026. You can find migration resources here.

Requirements

Before you start you’ll need:

An AWS account (with the AWS_ACCESS_KEY and AWS_SECRET_KEY)
A VPC that is routable from the internet. (Follow those instructions if you need to create one)
A SSH public key. (Follow those instructions if you need a new one)
The AWS CLI configured (run aws configure).
A Grafana instance with a Loki data source already configured, you can use GrafanaCloud free trial.

For the sake of simplicity we’ll use a Grafana Cloud Loki and Grafana instances, you can get a free account for this tutorial at [Grafana Cloud], but all the steps are the same if you’re running your own Open Source version of Loki and Grafana instances.

To make it easy to learn all the following instructions are manual, however in a real setup we recommend you to use provisioning tools such as Terraform, CloudFormation, Ansible or Chef.

Creating an EC2 instance

As a first step we’re going to import our SSH key to AWS so that we can SSH to our future EC2 instance, let’s run our first command:

aws ec2 import-key-pair --key-name "promtail-ec2" --public-key-material fileb://~/.ssh/id_rsa.pub

Next we’re going to create a security group, make sure to note the group id, we’ll need it for the following command:

aws ec2 create-security-group --group-name promtail-ec2  --description "promtail on ec2" --vpc-id vpc-668d120f
{
    "GroupId": "sg-02c489bbdeffdca1d"
}

Now let’s authorize inbound access for SSH and Promtail server:

aws ec2 authorize-security-group-ingress --group-id sg-02c489bbdeffdca1d --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id sg-02c489bbdeffdca1d --protocol tcp --port 3100 --cidr 0.0.0.0/0

Note
You don’t need to open those ports to all IPs as shown above you can use your own IP range.

We’re going to create an Amazon Linux 2 instance as this is one of the most popular but feel free to use the AMI of your choice.

To create the instance use the following command, make sure to note the instance id:

aws ec2 run-instances --image-id ami-016b213e65284e9c9 --count 1 --instance-type t2.micro --key-name promtail-ec2 --security-groups promtail-ec2

To make it more interesting later let’s tag (Name=promtail-demo) our instance:

aws ec2 create-tags --resources i-041b0be05c2d5cfad --tags Key=Name,Value=promtail-demo

Note
Tags enable you to categorize your AWS resources in different ways, for example, by purpose, owner, or environment. This is useful when you have many resources of the same type—you can quickly identify a specific resource based on the tags that you’ve assigned to it. You’ll see later, Promtail can transform those tags into [Loki labels][labels].

Finally let’s grab the public DNS of our instance:

aws ec2 describe-instances --filters "Name=tag:Name,Values=promtail-demo" --query "Reservations[].Instances[].NetworkInterfaces[].Association.PublicDnsName"

and start an SSH session:

ssh ec2-user@ec2-13-59-62-37.us-east-2.compute.amazonaws.com

Setting up Promtail

First let’s make sure we’re running as root by using sudo -s. Next we’ll download, install and give executable right to Promtail.

mkdir /opt/promtail && cd /opt/promtail
curl -O -L "https://github.com/grafana/loki/releases/download/v2.0.0/promtail-linux-amd64.zip"
unzip "promtail-linux-amd64.zip"
chmod a+x "promtail-linux-amd64"

Now we’re going to download the Promtail configuration file below and edit it, don’t worry we will explain what those means. The file is also available as a gist at cyriltovena/promtail-ec2.yaml.

curl https://raw.githubusercontent.com/grafana/loki/main/docs/sources/send-data/promtail/cloud/ec2/promtail-ec2.yaml > ec2-promtail.yaml
vi ec2-promtail.yaml

server:
  http_listen_port: 3100
  grpc_listen_port: 0

clients:
  - url: https://<user id>:<api secret>@logs-prod-us-central1.grafana.net/loki/api/v1/push

positions:
  filename: /opt/promtail/positions.yaml

scrape_configs:
  - job_name: ec2-logs
    ec2_sd_configs:
      - region: us-east-2
        access_key: REDACTED
        secret_key: REDACTED
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: name
        action: replace
      - source_labels: [__meta_ec2_instance_id]
        target_label: instance
        action: replace
      - source_labels: [__meta_ec2_availability_zone]
        target_label: zone
        action: replace
      - action: replace
        replacement: /var/log/**.log
        target_label: __path__
      - source_labels: [__meta_ec2_private_dns_name]
        regex: "(.*)"
        target_label: __host__

The server section indicates Promtail to bind his http server to 3100. Promtail serves HTTP pages for troubleshooting service discovery and targets.

The clients section allow you to target your loki instance, if you’re using GrafanaCloud simply replace <user id> and <api secret> with your credentials. Otherwise just replace the whole URL with your custom Loki instance.(e.g http://my-loki-instance.my-org.com/loki/api/v1/push)

Promtail uses the same Prometheus scrape_configs. This means if you already own a Prometheus instance the config will be very similar and easy to grasp.

Since we’re running on AWS EC2 we want to uses EC2 service discovery, this will allows us to scrape metadata about the current instance (and even your custom tags) and attach those to our logs. This way managing and querying on logs will be much easier.

Make sure to replace accordingly you current region, access_key and secret_key, alternatively you can use an AWS Role ARN, for more information about this, see documentation for ec2_sd_config.

Finally the relabeling_configs section has three purposes:

Selecting the labels discovered you want to attach to your targets. In our case here, we’re keeping instance_id as instance, the tag Name as name and the zone of the instance. Make sure to check out the Prometheus ec2_sd_config documentation for the full list of available labels.
Choosing where Promtail should find log files to tail, in our example we want to include all log files that exist in /var/log using the glob /var/log/**.log. If you need to use multiple glob, you can simply add another job in your scrape_configs.
Ensuring discovered targets are only for the machine Promtail currently runs on. This is achieved by adding the label __host__ using the incoming metadata __meta_ec2_private_dns_name. If it doesn’t match the current HOSTNAME environment variable, the target will be dropped. If __meta_ec2_private_dns_name doesn’t match your instance’s hostname (on EC2 Windows instance for example, where it is the IP address and not the hostname), you can hardcode the hostname at this stage, or check if any of the instances tag contain the hostname (__meta_ec2_tag_<tagkey>: each tag value of the instance)

Alright we should be ready to fire up Promtail, we’re going to run it using the flag --dry-run. This is perfect to ensure everything is correctly, specially when you’re still playing around with the configuration. Don’t worry when using this mode, Promtail won’t send any logs and won’t remember any file positions.

 ./promtail-linux-amd64 -config.file=./ec2-promtail.yaml --dry-run

If everything is going well Promtail should print out log lines with their labels discovered instead of sending them to Loki, like shown below:

2020-07-08T14:51:38-0700	{filename="/var/log/cloud-init.log", instance="i-041b0be05c2d5cfad", name="promtail-demo", zone="us-east-2c"}	Jul 07 21:37:24 cloud-init[3035]: util.py[DEBUG]: loaded blob returned None, returning default.

Don’t hesitate to edit the your config file and start Promtail again to try your config out.

If you want to see existing targets and available labels you can reach Promtail server using the public dns assigned to your instance:

open http://ec2-13-59-62-37.us-east-2.compute.amazonaws.com:3100/

For example the page below is the service discovery page. It shows you all discovered targets, with their respective available labels and the reason it was dropped if it was the case.

This page is really useful to understand what labels are available to forward with the relabeling configuration but also why Promtail is not scraping your target.

Configuring Promtail as a service

Now that we have correctly configured Promtail. We usually want to make sure it runs as a systemd service, so it can automatically restart on failure or when the instance restart.

Let’s create a new service using vim /etc/systemd/system/promtail.service and copy the service definition below:

[Unit]
Description=Promtail

[Service]
User=root
WorkingDirectory=/opt/promtail/
ExecStartPre=/bin/sleep 30
ExecStart=/opt/promtail/promtail-linux-amd64 --config.file=./ec2-promtail.yaml
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Let’s reload the systemd, enable then start the Promtail service:

systemctl daemon-reload
systemctl enable promtail.service
systemctl start promtail.service

You can verify that the service run correctly using the following command:

systemctl status promtail.service -l

● promtail.service - Promtail
   Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-07-08 15:48:57 UTC; 4s ago
 Main PID: 2732 (promtail-linux-)
   CGroup: /system.slice/promtail.service
           └─2732 /opt/promtail/promtail-linux-amd64 --config.file=./ec2-promtail.yaml

Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal systemd[1]: Started Promtail.
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal systemd[1]: Starting Promtail...
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=warn ts=2020-07-08T15:48:57.559085451Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.559869071Z caller=server.go:179 http=[::]:3100 grpc=[::]:35127 msg="server listening on addresses"
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.56029474Z caller=main.go:67 msg="Starting Promtail" version="(version=1.6.0, branch=HEAD, revision=12c7eab8)"

You can now verify in Grafana that Loki has correctly received your instance logs by using the LogQL query {zone="us-east-2"}.

Sending systemd logs

Just like we did with Promtail, you’ll most likely manage your applications with systemd which usually store applications logs in journald. Promtail actually support scraping logs from journald so let’s configure it.

We will edit our previous config (vi ec2-promtail.yaml) and add the following block in the scrape_configs section.

  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

Note that you can use relabeling to convert systemd labels to match what you want. Finally make sure that the path of journald logs is correct, it might be different on some systems.

Note
You can download the final config example from our [GitHub repository][final config].

That’s it, save the config and you can reboot the machine (or simply restart the service systemctl restart promtail.service).

Let’s head back to Grafana and verify that your Promtail logs are available in Grafana by using the LogQL query {unit="promtail.service"} in Explore. Finally make sure to checkout live tailing to see logs appearing as they are ingested in Loki.

Troubleshooting Promtail on AWS EC2

If you encounter the error error="filetarget.fsnotify.NewWatcher: too many open files" in standalone EC2, you can resolve it by adding the LimitNOFILE setting to your promtail.service file as follows:

# /etc/systemd/system/promtail.service
[Service]
...
LimitNOFILE=500000

The complete promtail.service file should look like this:

# /etc/systemd/system/promtail.service
[Unit]
Description=Promtail

[Service]
User=root
WorkingDirectory=/opt/promtail/
ExecStartPre=/bin/sleep 30
ExecStart=/opt/promtail/promtail-linux-amd64 --config.file=./ec2-promtail.yaml
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
LimitNOFILE=500000

[Install]
WantedBy=multi-user.target

Reload and restart the promtail daemon to apply the changed systemd settings.

systemctl daemon-reload
systemctl restart promtail.service

Continue to monitor whether the too many open files error occurs again.

watch -d -n2 "systemctl status promtail -l"

Run the Promtail client on AWS ECS

Sat, 11 Apr 2026 00:53:11 +0000

Run the Promtail client on AWS ECS

ECS is the fully managed container orchestration service by Amazon. Combined with Fargate you can run your container workload without the need to provision your own compute resources. In this tutorial we will see how you can leverage Firelens an AWS log router to forward all your logs and your workload metadata to a Grafana Loki instance.

Caution
Promtail has been deprecated and is in Long-Term Support (LTS) through February 28, 2026. Promtail will reach an End-of-Life (EOL) on March 2, 2026. You can find migration resources here.

After this tutorial you will able to query all your logs in one place using Grafana.

Requirements

Before we start you’ll need:

The AWS CLI configured (run aws configure).
A Grafana instance with a Loki data source already configured, you can use GrafanaCloud free trial.
A Subnet in VPC that is routable from the internet. (Follow those instructions if you need to create one).
A Security group of your choice for your containers. (Follow those instructions if you need to create one).

For the sake of simplicity we’ll use a GrafanaCloud Loki and Grafana instances, you can get an free account for this tutorial on our website, but all the steps are the same if you’re running your own Open Source version of Loki and Grafana instances.

Setting up the ECS cluster

To run containers with ECS you need an ECS cluster, we’ll use a Fargate cluster, but if you prefer to use an EC2 cluster all the given steps are still applicable.

Let’s create the cluster with awscli:

aws ecs create-cluster --cluster-name ecs-firelens-cluster

We will also need an IAM Role to run containers with, let’s create a new one and authorize ECS to endorse this role.

Note
You might already have this ecsTaskExecutionRole role in your AWS account if that’s the case you can skip this step.

curl https://raw.githubusercontent.com/grafana/loki/main/docs/sources/send-data/promtail/cloud/ecs/ecs-role.json > ecs-role.json
aws iam create-role --role-name ecsTaskExecutionRole  --assume-role-policy-document file://ecs-role.json

{
    "Role": {
        "Path": "/",
        "RoleName": "ecsTaskExecutionRole",
        "RoleId": "AROA5FW5RZWLXFPU656SQ",
        "Arn": "arn:aws:iam::0000000000:role/ecsTaskExecutionRole",
        "CreateDate": "2020-07-09T14:51:49+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": [
                            "ecs-tasks.amazonaws.com"
                        ]
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Note down the ARN of this new role, we’ll use it later to create an ECS task.

Finally we’ll give the ECS task execution policy AmazonECSTaskExecutionRolePolicy to the created role, this will allows us to manage logs with Firelens:

aws iam attach-role-policy --role-name ecsTaskExecutionRole --policy-arn "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"

Creating your task definition

Amazon Firelens is a log router (usually fluentd or fluentbit) you run along the same task definition next to your application containers to route their logs to Loki.

In this example we will use fluentbit with the fluentbit output plugin installed but if you prefer fluentd make sure to check the fluentd output plugin documentation.

Note
We recommend you to use [fluentbit][fluentbit] as it’s less resources consuming than [fluentd][fluentd].

Our task definition will be made of two containers, the Firelens log router to send logs to Loki (log_router) and a sample application to generate log with (sample-app).

Let’s download the task definition, we’ll go through the most important parts.

curl https://raw.githubusercontent.com/grafana/loki/main/docs/sources/send-data/promtail/cloud/ecs/ecs-task.json > ecs-task.json

 {
    "essential": true,
    "image": "grafana/fluent-bit-plugin-loki:2.9.3-amd64",
    "name": "log_router",
    "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
            "enable-ecs-log-metadata": "true"
        }
    },
    "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "firelens-container",
            "awslogs-region": "us-east-2",
            "awslogs-create-group": "true",
            "awslogs-stream-prefix": "firelens"
        }
    },
    "memoryReservation": 50
},

The log_router container image is the Fluent bit Loki docker image which contains the Loki plugin pre-installed. As you can see the firelensConfiguration type is set to fluentbit and we’ve also added options to enable ECS log metadata. This will be useful when querying your logs with Loki LogQL label matchers.

Note
The logConfiguration is mostly there for debugging the fluent-bit container, but feel free to remove that part when you’re done testing and configuring.

 {
    "command": [
        "/bin/sh -c \"while true; do sleep 15 ;echo hello_world; done\""
    ],
    "entryPoint": ["sh","-c"],
    "essential": true,
    "image": "alpine:3.13",
    "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
            "Name": "loki",
            "Host": "<grafanacloud host>",
            "Http_User": "<userid>", 
            "Labels": "{job=\"firelens\"}",
            "RemoveKeys": "container_id,ecs_task_arn",
            "LabelKeys": "container_name,ecs_task_definition,source,ecs_cluster",
            "LineFormat": "key_value"
        },
        "secretOptions": [{
            "name": "Http_Passwd",
            "valueFrom": "data.aws_secretsmanager_secret.grafana_cloud_loki_http_password.id"
        }]
    },
    "name": "sample-app"
}

The second container is our sample-app, a simple alpine container that prints to stdout welcoming messages. To send those logs to Loki, we will configure this container to use the log driver awsfirelens.

Go ahead and replace the Host and HTTP_User property with your GrafanaCloud credentials, you can find them in your account in the Loki instance page. If you’re running your own Loki instance replace completely the URL (for example, http://my-loki.com:3100/loki/api/v1/push).

We include plain text credentials in options for simplicity. However, this exposes credentials in your ECS task definition and in any version-controlled configuration. Mitigate this issue by using a secret store such as AWS Secrets Manager, combined with the secretOptions configuration option for injecting sensitive data in a log configuration.

All options of the logConfiguration will be automatically translated into fluentbit output. For example, the above options will produce this fluent bit OUTPUT config section:

[OUTPUT]
    Name grafana-loki
    Match awsfirelens*
    Url https://<userid>:<grafancloud apikey>@logs-prod-us-central1.grafana.net/loki/api/v1/push
    Labels {job="firelens"}
    RemoveKeys container_id,ecs_task_arn
    LabelKeys container_name,ecs_task_definition,source,ecs_cluster
    LineFormat key_value

This OUTPUT config will forward logs to GrafanaCloud Loki, to learn more about those options make sure to read the fluentbit output plugin documentation. We’ve kept some interesting and useful labels such as container_name, ecs_task_definition , source and ecs_cluster but you can statically add more via the Labels option.

Note
If you want run multiple containers in your task, all of them needs a logConfiguration section, this give you the opportunity to add different labels depending on the container.

{
    "containerDefinitions": [
     ...
    ],
    "cpu": "256",
    "executionRoleArn": "arn:aws:iam::00000000:role/ecsTaskExecutionRole",
    "family": "loki-fargate-task-definition",
    "memory": "512",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ]
}

Finally, you need to replace the executionRoleArn with the ARN of the role we created in the first section.

Once you’ve finished editing the task definition we can then run the command below to create the task:

aws ecs register-task-definition --region us-east-2 --cli-input-json  file://ecs-task.json

Now let’s create and start a service.

Running your service

To run the service you need to provide the task definition name loki-fargate-task-definition:1 which is the combination of task family plus the task revision :1. You also need your own subnet and security group, you can replace respectively subnet-306ca97d and sg-02c489bbdeffdca1d in the command below and start the your service:

aws ecs create-service --cluster ecs-firelens-cluster \
--service-name firelens-loki-fargate \
--task-definition loki-fargate-task-definition:1 \
--desired-count 1 --region us-east-2 --launch-type "FARGATE" \
--network-configuration "awsvpcConfiguration={subnets=[subnet-306ca97d],securityGroups=[sg-02c489bbdeffdca1d],assignPublicIp=ENABLED}"

Note
Make sure public (assignPublicIp) is enabled otherwise ECS won’t connect to the internet and you won’t be able to pull external docker images.

You can now access the ECS console and you should see your task running. Now let’s open Grafana and use explore with the Loki data source to explore our task logs. Enter the query {job="firelens"} and you should see our sample-app logs showing up as shown below:

Using the Log Labels dropdown you should be able to discover your workload via the ECS metadata, which is also visible if you expand a log line.

That’s it. Make sure to checkout LogQL to learn more about Loki powerful query language.

Run the Promtail client on AWS EKS

Sat, 11 Apr 2026 00:53:11 +0000

Run the Promtail client on AWS EKS

In this tutorial we’ll see how to set up Promtail on EKS. Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed Kubernetes service, using Promtail we’ll get full visibility into our cluster logs. We’ll start by forwarding pods logs then nodes services and finally Kubernetes events.

Caution
Promtail has been deprecated and is in Long-Term Support (LTS) through February 28, 2026. Promtail will reach an End-of-Life (EOL) on March 2, 2026. You can find migration resources here.

After this tutorial you will able to query all your logs in one place using Grafana.

Requirements

Before we start you’ll need:

The AWS CLI configured (run aws configure).
kubectl and eksctl installed.
A Grafana instance with a Grafana Loki data source already configured, you can use GrafanaCloud free trial.

Setting up the cluster

In this tutorial we’ll use eksctl, a simple command line utility for creating and managing Kubernetes clusters on Amazon EKS. AWS requires creating many resources such as IAM roles, security groups and networks, by using eksctl all of this is simplified.

Note
We’re not going to use a Fargate cluster. Do note that if you want to use Fargate daemonset are not allowed, the only way to ship logs with EKS Fargate is to run a fluentd or fluentbit or Promtail as a sidecar and tee your logs into a file. For more information on how to do so, you can read this [blog post][blog ship log with fargate].

eksctl create cluster --name loki-promtail --managed

This usually takes about 15 minutes. When this is finished you should have kubectl context configured to communicate with your newly created cluster. To verify, run the following command:

kubectl version

You should see output similar to the following:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-04T15:01:15Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-fd1ea7", GitCommit:"fd1ea7c64d0e3ccbf04b124431c659f65330562a", GitTreeState:"clean", BuildDate:"2020-05-28T19:06:00Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Adding Promtail DaemonSet

To ship all your pods logs we’re going to set up Promtail as a DaemonSet in our cluster. This means it will run on each nodes of the cluster, we will then configure it to find the logs of your containers on the host.

What’s nice about Promtail is that it uses the same service discovery as Prometheus, you should make sure the scrape_configs of Promtail matches the Prometheus one. Not only this is simpler to configure, but this also means Metrics and Logs will have the same metadata (labels) attached by the Prometheus service discovery. When querying Grafana you will be able to correlate metrics and logs very quickly, you can read more about this on our blogpost.

Let’s add the Loki repository and list all available charts. To add the repo, run the following command:

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade -i promtail grafana/promtail

You should see the following message.

"loki" has been added to your repositories

To list the available charts, run the following command:

helm search repo

You should see output similar to the following:

NAME                   CHART VERSION   APP VERSION     DESCRIPTION
loki/fluent-bit 0.3.0           v1.6.0          Uses fluent-bit Loki go plugin for gathering lo...
loki/loki       0.31.0          v1.6.0          Loki: like Prometheus, but for logs.
loki/loki-stack 0.40.0          v1.6.0          Loki: like Prometheus, but for logs.
loki/promtail   0.24.0          v1.6.0          Responsible for gathering logs and sending them...

If you want to install Loki, Grafana, Prometheus and Promtail all together you can use the loki-stack chart, for now we’ll focus on Promtail. Let’s create a new helm value file, we’ll fetch the default one and work from there:

curl https://raw.githubusercontent.com/grafana/helm-charts/main/charts/promtail/values.yaml > values.yaml

First we’re going to tell Promtail to send logs to our Loki instance, the example below shows how to send logs to GrafanaCloud, replace your credentials. The default value will send to your own Loki and Grafana instance if you’re using the loki-chart repository.

loki:
  serviceName: "logs-prod-us-central1.grafana.net"
  servicePort: 443
  serviceScheme: https
  user: <userid>
  password: <grafancloud apikey>

Once you’re ready let’s create a new namespace monitoring and add Promtail to it. To create the namespace, run the following command:

kubectl create namespace monitoring

You should see the following message.

namespace/monitoring created

To add Promtail, run the following command:

helm install promtail --namespace monitoring loki/promtail -f values.yaml

You should see output similar to the following:

NAME: promtail
LAST DEPLOYED: Fri Jul 10 14:41:37 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Verify the application is working by running these commands:
  kubectl --namespace default port-forward daemonset/promtail 3101
  curl http://127.0.0.1:3101/metrics

Verify that Promtail pods are running. You should see only two since we’re running a two nodes cluster.

kubectl get -n monitoring pods

You should see output similar to the following:

NAME             READY   STATUS    RESTARTS   AGE
promtail-87t62   1/1     Running   0          35s
promtail-8c2r4   1/1     Running   0          35s

You can reach your Grafana instance and start exploring your logs. For example if you want to see all logs in the monitoring namespace use {namespace="monitoring"}, you can also expand a single log line to discover all labels available from the Kubernetes service discovery.

Fetching kubelet logs with systemd

So far we’re scrapings logs from containers, but if you want to get more visibility you could also scrape systemd logs from each of your machine. This means you can also get access to kubelet logs.

Let’s edit our values file again and extraScrapeConfigs to add the systemd job:

extraScrapeConfigs:
  - job_name: journal
    journal:
      path: /var/log/journal
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__hostname']
        target_label: 'hostname'

Feel free to change the relabel_configs to match what you would use in your own environnement.

Now we need to add a volume for accessing systemd logs:

extraVolumes:
  - name: journal
    hostPath:
      path: /var/log/journal

And add a new volume mount in Promtail:

extraVolumeMounts:
  - name: journal
    mountPath: /var/log/journal
    readOnly: true

Now that we’re ready we can update the Promtail deployment:

helm upgrade  promtail loki/promtail -n monitoring -f values.yaml

Let go back to Grafana and type in the query below to fetch all logs related to Volume from Kubelet:

{unit="kubelet.service"} |= "Volume"

Filter expressions are powerful in LogQL they help you scan through your logs, in this case it will filter out all your kubelet logs not having the Volume word in it.

The workflow is simple, you always select a set of labels matchers first, this way you reduce the data you’re planing to scan.(such as an application, a namespace or even a cluster). Then you can apply a set of filters to find the logs you want.

Promtail also supports syslog.

Adding Kubernetes events

Kubernetes Events (kubectl get events -n monitoring) are a great way to debug and troubleshoot your kubernetes cluster. Events contains information such as Node reboot, OOMKiller and Pod failures.

We’ll deploy a the eventrouter application created by Heptio which logs those events to stdout.

But first we need to configure Promtail, we want to parse the namespace to add it as a label from the content, this way we can quickly access events by namespace.

Let’s update our pipelineStages to parse logs from the eventrouter:

pipelineStages:
- docker:
- match:
    selector: '{app="eventrouter"}'
    stages:
    - json:
        expressions:
          namespace: event.metadata.namespace
    - labels:
        namespace: ""

Pipeline stages are great ways to parse log content and create labels (which are indexed), if you want to configure more of them, check out the pipeline documentation.

Now update Promtail again:

helm upgrade  promtail loki/promtail -n monitoring -f values.yaml

And deploy the eventrouter using:

kubectl create -f https://raw.githubusercontent.com/grafana/loki/main/docs/sources/send-data/promtail/cloud/eks/eventrouter.yaml

You should see output similar to the following:

serviceaccount/eventrouter created
clusterrole.rbac.authorization.k8s.io/eventrouter created
clusterrolebinding.rbac.authorization.k8s.io/eventrouter created
configmap/eventrouter-cm created
deployment.apps/eventrouter created

Let’s go in Grafana Explore and query events for our new monitoring namespace using {app="eventrouter",namespace="monitoring"}.

For more information about the eventrouter make sure to read our blog post from Goutham.

Conclusion

That’s it ! You can download the final and complete values.yaml if you need.

Your EKS cluster is now ready, all your current and future application logs will now be shipped to Loki with Promtail. You will also able to explore kubelet and Kubernetes events. Since we’ve used a DaemonSet you’ll automatically grab all your node logs as you scale them.

If you want to push this further you can check out Joe’s blog post on how to automatically create Grafana dashboard annotations with Loki when you deploy new Kubernetes applications.

If you need to delete the cluster simply run eksctl delete cluster --name loki-promtail

Run the Promtail client on Google Cloud Platform

Sat, 11 Apr 2026 00:53:11 +0000

Run the Promtail client on Google Cloud Platform

This document explains how one can setup Google Cloud Platform to forward its cloud resource logs from a particular GCP project into Google Pubsub topic so that is available for Promtail to consume.

Caution
Promtail has been deprecated and is in Long-Term Support (LTS) through February 28, 2026. Promtail will reach an End-of-Life (EOL) on March 2, 2026. You can find migration resources here.

This document assumes, that reader have gcloud installed and have the required permissions (as mentioned in Roles and Permission section).

There are two flavours of how to configure this:

Pull-based subscription: Promtail pulls log entries from a GCP PubSub topic
Push-based subscription: GCP sends log entries to a web server that Promtail listens

Overall, the setup between GCP, Promtail and Loki will look like the following:

Roles and Permission

The user should have following roles to complete the setup.

“roles/pubsub.editor”
“roles/logging.configWriter”

Setup Pubsub Topic

Google Pubsub Topic will act as the queue to persist log messages which then can be read from Promtail.

$ gcloud pubsub topics create $TOPIC_ID

For example:

$ gcloud pubsub topics create cloud-logs

Setup Log Router

We create a log sink to forward cloud logs into pubsub topic that we just created.

$ gcloud logging sinks create $SINK_NAME $SINK_LOCATION $OPTIONAL_FLAGS

For example:

$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \
--log-filter='resource.type=("gcs_bucket")' \
--description="Cloud logs"

The above command also adds log-filter option which represents what type of logs should get into the destination pubsub topic. For more information on adding log-filter refer this document

We cover more advanced log-filter below

Grant log sink the pubsub publisher role

Find the writer identity service account of the log sink just created:

gcloud logging sinks describe \
 --format='value(writerIdentity)' $SINK_NAME

For example:

gcloud logging sinks describe \
 --format='value(writerIdentity)' cloud-logs

Create an IAM policy binding to allow log sink to publish messages to the topic:

gcloud pubsub topics add-iam-policy-binding $TOPIC_ID \
--member=$WRITER_IDENTITY --role=roles/pubsub.publisher

For example:

gcloud pubsub topics add-iam-policy-binding cloud-logs \
--member=serviceAccount:pxxxxxxxxx-xxxxxx@gcp-sa-logging.iam.gserviceaccount.com --role=roles/pubsub.publisher

Create Pubsub subscription for Grafana Loki

Pull

We create subscription for the pubsub topic we create above and Promtail uses this subscription to consume the log messages.

$ gcloud pubsub subscriptions create cloud-logs --topic=$TOPIC_ID \
--ack-deadline=$ACK_DEADLINE \
--message-retention-duration=$RETENTION_DURATION \

e.g:

$ gcloud pubsub subscriptions create cloud-logs --topic=projects/my-project/topics/cloud-logs \
--ack-deadline=10 \
--message-retention-duration=7d

For more fine grained options, refer to the gcloud pubsub subscriptions --help

Push

Since GCP PubSub push subscriptions is a rather new service, In some cases, it is necessary to grant Google permission. First, check the date the GCP project was created on:

> gcloud projects describe $GCP_PROJECT_ID

createTime: 'some date'
...
projectId: $GCP_PROJECT_ID
projectNumber: '$GCP_PROJECT_NUMBER'

If the createTime is earlier than April 8, 2021, skip the following step. Otherwise, you need to grant the the iam.serviceAccountTokenCreator role to a Google-managed service account:

PUBSUB_SERVICE_ACCOUNT="service-${GCP_PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
 --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\
 --role='roles/iam.serviceAccountTokenCreator'

have configured Promtail with the GCP Logs Push target, hosted in an internet-facing and HTTPS enabled deployment, we can continue with creating the push subscription.

gcloud pubsub subscriptions create cloud-logs \
--topic=$TOPIC_ID \
--push-endpoint=$HTTPS_PUSH_ENDPOINT_URI

Setup using Terraform

You also have the option of creating the required resources for GCP Log collection with terraform. First, the following snippet will add the resources needed for both pull and push flavours.

How to use Terraform is outside the scope of this guide. You can find this tutorial on how to work with Terraform and GCP useful.

// Provider module
provider "google" {
  project = "$GCP_PROJECT_ID"
}

// Topic
resource "google_pubsub_topic" "main" {
  name = "cloud-logs"
}

// Log sink
variable "inclusion_filter" {
  type        = string
  description = "Optional GCP Logs query which can filter logs being routed to the pub/sub topic and promtail"
}

resource "google_logging_project_sink" "main" {
  name                   = "cloud-logs"
  destination            = "pubsub.googleapis.com/${google_pubsub_topic.main.id}"
  filter                 = var.inclusion_filter
  unique_writer_identity = true
}

resource "google_pubsub_topic_iam_binding" "log-writer" {
  topic = google_pubsub_topic.main.name
  role  = "roles/pubsub.publisher"
  members = [
    google_logging_project_sink.main.writer_identity,
  ]
}

Then, another snippet needs to be added depending on whether the pull or push flavour is chosen.

Pull

The following snippet configures a subscription to the pub/sub topic which Promtail will subscribe to.

// Subscription
resource "google_pubsub_subscription" "main" {
  name  = "cloud-logs"
  topic = google_pubsub_topic.main.name
}

Then, to create the new resources run the snippet below after filling in the required variables.

terraform apply \
    -var="inclusion_filter=<GCP Logs query of what logs to include>"

Push

The following snippet configures a push subscription to the pub/sub topic which will forward logs to Promtail.

// Subscription
variable "push_endpoint_url" {
  type        = string
  description = "Public URL where Promtail is hosted."
}

resource "google_pubsub_subscription" "main" {
  name  = "cloud-logs"
  topic = google_pubsub_topic.main.name
  push_config {
    push_endpoint = var.push_endpoint_url
    attributes = {
      x-goog-version = "v1"
    }
  }
}

To create the new resources, run the snippet below after filling in the required variables.

terraform apply \
    -var="push_endpoint_url=<Promtail public URL>" \
    -var="inclusion_filter=<GCP Logs query of what logs to include>"

ServiceAccount for Promtail

We need a service account with the following permissions:

pubsub.subscriber

This enables Promtail to read log entries from the pubsub subscription created before.

You can find an example for Promtail scrape config for gcplog here

If you are scraping logs from multiple GCP projects, then this serviceaccount should have above permissions in all the projects you are tyring to scrape.

Operations

Sometimes, you may wish to clear the pending pubsub queue containing logs.

These messages stays in Pubsub Subscription until they’re acknowledged. The following command removes log messages without needing to be consumed via Promtail or any other pubsub consumer.

gcloud pubsub subscriptions seek <subscription-path> --time=<yyyy-mm-ddThh:mm:ss>

To delete all the old messages until now, set --time to current time.

gcloud pubsub subscriptions seek projects/my-project/subscriptions/cloud-logs --time=$(date +%Y-%m-%dT%H:%M:%S)

Advanced log filter

So far we’ve covered admitting GCS bucket logs into Grafana Loki, but often one may need to add multiple cloud resource logs and may also need to exclude unnecessary logs. The following is a more complex example.

We use the log-filter option to include logs and the exclusion option to exclude specific logs.

Use Case

Include the following cloud resource logs:

GCS bucket
Kubernetes
IAM
HTTP Load balancer

We also exclude specific HTTP load balancer logs based on payload and status code.

$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \
--log-filter='resource.type=("gcs_bucket OR k8s_cluster OR service_account OR iam_role OR api OR audited_resource OR http_load_balancer")' \
--description="Cloud logs" \
--exclusion='name=http_load_balancer,filter=<<EOF
resource.type="http_load_balancer"
(
	(
		jsonPayload.statusDetails=("byte_range_caching" OR "websocket_closed")
	)
		OR
	(
		http_request.status=(101 OR 206)
	)
)
EOF