Loki:NamespaceLogs

Dashboard

Loki dashboards for promtail metrics
Last updated: a month ago

Start with Grafana Cloud and the new FREE tier. Includes 10K series Prometheus or Graphite Metrics and 50gb Loki Logs

Downloads: 7

Reviews: 0

    grafana + MySQL + InfluxDB+ telegraf + Loki + promtail

    telegraf UI telegraf UI loki UI

    My Dashbards

    After installing git monitoring-k8s,All settings are complete.
    If you want to use git monitoring-k8s without installing it, delete the regular expression described in the variable.
    git の monitoring-k8s をインストールすると、全ての設定が完了します。
    git の monitoring-k8s インストールしないで利用する場合は、変数に記載している正規表現を削除してください。
    Loki:AccessLogs
    Loki:NamespaceLogs
    Loki:SystemLogs
    Telegraf:KubernetesResources
    Telegraf:SystemResources

    setup

    git clone https://github.com/AbeYuki/monitoring-k8s.git
    

    はじめに

    README.md ファイルがある場所へ移動
    シークレットを含んだファイル(★add, ★fix)の追加、修正を行い deploy する流れ
    cat リダイレクトでファイル作成例としているが、エディタでの作成を推奨

    .
    ├── README.md
    ├── grafana
    │   ├── deployment-backend-grafana-db01.yaml
    │   ├── deployment-frontend-grafana-app01.yaml
    │   └── grafana.ini ★add
    ├── influxdb
    │   ├── configmap-influxdb-init01.yaml
    │   └── deployment-backend-influxdb-db01.yaml
    ├── kustomization.yaml ★fix
    ├── label-transformer.yaml
    ├── loki
    │   ├── config.yaml
    │   └── deployment-frontend-loki-app01.yaml
    ├── namespace.yaml
    ├── password.txt ★add
    ├── promtail
    │   ├── config.yaml
    │   └── daemonset-backend-agent01.yaml
    ├── rbac
    │   ├── rbac-influxdb.yaml
    │   ├── rbac-prometheus.yaml
    │   ├── rbac-promtail.yaml
    │   └── rbac-telegraf.yaml
    ├── telegraf
    │   ├── daemonset-backend-agent01.yaml
    │   └── telegraf.conf ★add
    └── token.txt ★add
    

    kustomize.yaml setup

    secretGenerator で作成するパスワード、トークンファイル作成

    echo -n 'password' > password.txt
    
    echo -n 'token' > token.txt
    

    grafana setup

    kustomization.yaml で指定したパスワードに修正し、grafana/grafana.ini ファイル作成

    [database]  
      password = 修正
      mysql://grafana:修正@monitoring-backend-grafana-db01-001:3306/grafana
    
    [session]  
      provider_config = `grafana:修正@tcp(monitoring-backend-grafana-db01-001:3306)/grafana` 
    

    password に "#" または ";" が含まれている場合は三重引用符にする必要がある
    例)

    [database]
      password = #password; -> 誤
    [database]
      password = """#password;""" -> 正
    
    cat <<'EOF'> grafana/grafana.ini
    [server]
      protocol = http
      http_port = 3000
    [database]
      type = mysql
      host = monitoring-backend-grafana-db01-001:3306
      name = grafana
      user = grafana
      password = password
      ssl_mode = disable
      url = mysql://grafana:password@monitoring-backend-grafana-db01-001:3306/grafana
    [session]
      provider_config = `grafana:password@tcp(monitoring-backend-grafana-db01-001:3306)/grafana` 
      provider = mysql
    [analytics]
      reporting_enabled = false
      check_for_updates = true
    [log]
      mode = console
      level = info
    [paths]
      data         = /var/lib/grafana/data
      logs         = /var/log/grafana
      plugins      = /var/lib/grafana/plugins
      provisioning = /etc/grafana/provisioning
    [unified_alerting]
      enabled = true
    [alerting]
    [annotations.api]
    [annotations.dashboard]
    [annotations]
    [auth.anonymous]
    [auth.azuread]
    [auth.basic]
    [auth.generic_oauth]
    [auth.github]
    [auth.gitlab]
    [auth.google]
    [auth.grafana_com]
    [auth.jwt]
    [auth.ldap]
    [auth.okta]
    [auth.proxy]
    [auth]
    [aws]
    [azure]
    [dashboards]
    [dataproxy]
    [datasources]
    [date_formats]
    [emails]
    [enterprise]
    [explore]
    [expressions]
    [external_image_storage.azure_blob]
    [external_image_storage.gcs]
    [external_image_storage.local]
    [external_image_storage.s3]
    [external_image_storage.webdav]
    [external_image_storage]
    [feature_toggles]
    [geomap]
    [grafana_com]
    [live]
    [log.console]
    [log.file]
    [log.frontend]
    [log.syslog]
    [metrics.environment_info]
    [metrics.graphite]
    [metrics]
    [panels]
    [plugin.grafana-image-renderer]
    [plugins]
    [quota]
    [remote_cache]
    [rendering]
    [security]
    [smtp]
    [snapshots]
    [tracing.jaeger]
    [unified_alerting]
    [users]
    EOF
    

    telegraf setup

    kustomization.yaml で指定したトークンに修正、disk 等の監視対象の調整を行い telegraf/telegraf.conf ファイル作成

    [[outputs.influxdb_v2]]  
      token = "修正"  
    [[inputs.disk]]  
      fstype = [ "ext4", "xfs" ]  
      path = [ "/", "/backup", "/var/lib/longhorn" ]  
    
    cat <<'EOF'> telegraf/telegraf.conf
    [agent]
      interval = "60s"
      round_interval = true
      metric_batch_size = 1000
      metric_buffer_limit = 10000
      collection_jitter = "0s"
      flush_interval = "10s"
      flush_jitter = "0s"
      precision = ""
      hostname = "$HOSTNAME"
      omit_hostname = false
    [[outputs.influxdb_v2]]
      urls = ["http://monitoring-backend-influxdb-db01-001:8086"]
      token = "修正"
      organization = "monitoirng"
      bucket = "monitoring"
      timeout = "5s"    
    [[inputs.cpu]]
      percpu = true
      totalcpu = true
      fielddrop = ["time_*"]
    [[inputs.system]]
    [[inputs.disk]]
      ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
      mount_points = ["/", "/backup", "/var/lib/longhorn"]
    [inputs.disk.tagpass]
      fstype = [ "ext4", "xfs" ]
      path = [ "/", "/backup", "/var/lib/longhorn" ]
    [[inputs.diskio]]
      devices = ["sd*"]
    [[inputs.kernel]]
    [[inputs.mem]]
    [[inputs.processes]]
    [[inputs.swap]]
    [[inputs.system]]
    [[inputs.net]]
    [[inputs.netstat]]
    [[inputs.interrupts]]
    [[inputs.linux_sysctl_fs]]
    [[inputs.docker]]
      endpoint = "unix:///var/run/docker.sock"
    [[inputs.kubernetes]]
      url = "https://$HOSTIP:10250"
      bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
      insecure_skip_verify = true
    EOF
    

    kustomization.yaml の patchesStrategicMerge を修正して storageclass,resources の調整

    patchesStrategicMerge:
    - |-
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: frontend-grafana-app01
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 調整
        storageClassName: 調整
    

    docker.sock の権限変更

    docker metrics を収集する場合、 docker.sock の権限を 666 に変更する

    sudo chmod 666 /var/run/docker.sock
    
    vagrant@ubuntu2004:~$ ls -l /var/run/docker.sock
    srw-rw---- 1 root docker 0 May  9 05:29 /var/run/docker.sock
    vagrant@ubuntu2004:~$ sudo chmod 666 /var/run/docker.sock
    vagrant@ubuntu2004:~$ ls -l /var/run/docker.sock
    srw-rw-rw- 1 root docker 0 May  9 05:29 /var/run/docker.sock
    vagrant@ubuntu2004:~$ 
    

    deploy

    deploy namespace

    kubectl apply -f namespace.yaml
    

    deploy resource

    kubectl apply -k ./
    

    Grafana datasource setting

    Influxdb setting

    • Query Language
      • Flux
    • url
      • http://monitoring-backend-influxdb-db01-001:8086
    • Access
      • Server(default)
    • InfluxDB Details
      • Organization
        • monitoring
      • Token
        • token
      • Default Bucket
        • monitoring

    datasource-influxdb

    Loki setting

    • HTTP
      • URL
        • http://monitoring-frontend-loki-app01-001.monitoring.svc.cluster.local:3100

    datasource-influxdb

    Collector Configuration Details

    server:
      http_listen_port: 9080
      grpc_listen_port: 0
    positions:
      filename: /tmp/positions.yaml
    clients:
      - url: http://monitoring-frontend-loki-app01-001.monitoring.svc.cluster.local:3100/loki/api/v1/push
    scrape_configs:
    - job_name: kubernetes-pods-app
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - target_label: __service__
        source_labels: [__meta_kubernetes_pod_label_app]
      - target_label: __host__
        source_labels: [__meta_kubernetes_pod_node_name]
      - action: replace
        separator: /
        replacement: $1
        target_label: job
        source_labels: [__meta_kubernetes_namespace, __service__]
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        target_label: pod_name
        source_labels: [__meta_kubernetes_pod_name]
      - action: replace
        target_label: container_name
        source_labels: [__meta_kubernetes_pod_container_name]
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - target_label: __path__
        separator: /
        replacement: /var/log/pods/*$1/*.log
        source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
    - job_name: system
      static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
    - job_name: journal
      journal:
        max_age: 12h
        path: /var/log/journal
        labels:
          job: system-journal
      relabel_configs:
        - source_labels: ['__journal__systemd_unit']
          target_label: 'unit'
        - source_labels:
          - __journal__hostname
          target_label: nodename
        - source_labels:
          - __journal_syslog_identifier
          target_label: syslog_identifier
    
    Get this dashboard:
    16227
    Dependencies: