InstallationSimple scalable deployment with Helm

Simple scalable deployment of Grafana Loki with Helm

This Helm installation runs the Grafana Loki cluster in simple scalable deployment mode within a Kubernetes cluster.

Prerequisites

Deploy Loki to the Kubernetes cluster

  1. Add Loki’s chart repository to Helm:

    helm repo add grafana https://grafana.github.io/helm-charts
    
  2. Update the chart repository:

    helm repo update
    
  3. Deploy the Loki cluster using one of these commands.

    • Deploy with the default configuration:

      helm upgrade --install loki grafana/loki-simple-scalable
      
    • Deploy with the default configuration in a custom Kubernetes cluster namespace:

      helm upgrade --install loki --namespace=loki grafana/loki-simple-scalable
      
    • Deploy with added custom configuration:

      helm upgrade --install loki grafana/loki-simple-scalable --set "key1=val1,key2=val2,..."
      

Deploy Grafana to your Kubernetes cluster

Install Grafana on your Kubernetes cluster with Helm:

helm install loki-grafana grafana/grafana

To get the admin password for the Grafana pod, run the following command:

kubectl get secret --namespace <YOUR-NAMESPACE> loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

To access the Grafana UI, run the following command:

kubectl port-forward --namespace <YOUR-NAMESPACE> service/loki-grafana 3000:80

Navigate to http://localhost:3000 and login with admin and the password output above. Then follow the instructions for adding the Loki Data Source, using the URL http://<helm-installation-name>-gateway.<namespace>.svc.cluster.local:3100/ for Loki, with <helm-installation-name> and <namespaced> replaced with the correct values for your deployment.

Run Loki behind HTTPS Ingress

If Loki and Promtail are deployed on different clusters, you can add an Ingress in front of Loki. By adding a certificate, you create an HTTPS endpoint. For extra security you can also enable Basic Authentication on Ingress.

In the Promtail configuration, set the following values to communicate using HTTPS and basic authentication:

loki:
  serviceScheme: https
  user: user
  password: pass

Sample Helm template for Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: {{ .Values.ingress.class }}
    ingress.kubernetes.io/auth-type: "basic"
    ingress.kubernetes.io/auth-secret: {{ .Values.ingress.basic.secret }}
  name: loki
spec:
  rules:
  - host: {{ .Values.ingress.host }}
    http:
      paths:
      - backend:
          serviceName: loki
          servicePort: 3100
  tls:
  - secretName: {{ .Values.ingress.cert }}
    hosts:
    - {{ .Values.ingress.host }}

Run Promtail with syslog support

In order to receive and process syslog messages in Promtail, the following changes will be necessary:

  • Review the Promtail syslog-receiver configuration documentation

  • Configure the Promtail Helm chart with the syslog configuration added to the extraScrapeConfigs section and associated service definition to listen for syslog messages. For example:

    extraScrapeConfigs:
      - job_name: syslog
        syslog:
          listen_address: 0.0.0.0:1514
          labels:
            job: "syslog"
      relabel_configs:
        - source_labels: ['__syslog_message_hostname']
          target_label: 'host'
    syslogService:
      enabled: true
      type: LoadBalancer
      port: 1514
    

Run Promtail with systemd-journal support

In order to receive and process syslog message into Promtail, the following changes will be necessary:

  • Review the Promtail systemd-journal configuration documentation

  • Configure the Promtail Helm chart with the systemd-journal configuration added to the extraScrapeConfigs section and volume mounts for the Promtail pods to access the log files. For example:

    # Add additional scrape config
    extraScrapeConfigs:
      - job_name: journal
        journal:
          path: /var/log/journal
          max_age: 12h
          labels:
            job: systemd-journal
        relabel_configs:
          - source_labels: ['__journal__systemd_unit']
            target_label: 'unit'
          - source_labels: ['__journal__hostname']
            target_label: 'hostname'
    
    # Mount journal directory into Promtail pods
    extraVolumes:
      - name: journal
        hostPath:
          path: /var/log/journal
    
    extraVolumeMounts:
      - name: journal
        mountPath: /var/log/journal
        readOnly: true