Analytics - NGINX / LOKI v2+ Data Source / Promtail v2+ Tool

<h2 id="revision---21st-june-2022">Revision - 21st June 2022</h2> <ul> <li>Updated visitor map to support week on week comparison and top country listing</li> </ul> <h2 id="revision---20th-february-2021">Revision - 20th February 2021</h2> <ul> <li>Changed overall dashboard look and initial display of panels.</li> </ul> <h2 id="revision---16th-february-2021">Revision - 16th February 2021</h2> <ul> <li>Last 30 day graph of unique visitors to site added</li> <li>Removed more browser and crawler bot statistics from unique visitors and other relevant panels</li> </ul> <h2 id="revision---12th-february-2021">Revision - 12th February 2021</h2> <ul> <li>Changed Last 7 day graph from &rsquo;new time series beta graph&rsquo; to standard graph display.</li> <li>Removed further browser and crawler bot statistics from relevant panel - to add these back in edit the panel and remove below from query on each panel.</li> </ul> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">| http_user_agent !~ &#34;.*bot.*&#34; | remote_addr !~ &#34;2001:4ca0:108:42::*&#34; | remote_addr !~ &#34;91.134.156.*&#34; | http_user_agent != &#34;worldping-api&#34; | request_uri !~ &#34;/wp-.*&#34; | request_uri !~ &#34;//wp-.*&#34; | request_uri !~ &#34;/*.wordfence.*&#34; | request_uri !~ &#34;/robots.txt&#34; | request_uri !~ &#34;/xmlrpc.php&#34;</code></pre> </div> </div> <h2 id="extension-of-original-dashboard">Extension of original dashboard</h2> <p><a href="/grafana/dashboards/12559?pg=dashboards&amp;plcmt=featured-main">https://grafana.com/grafana/dashboards/12559?pg=dashboards&amp;plcmt=featured-main</a></p> <h2 id="setup-of-nginx-promtail--loki-required">Setup of nginx, promtail &amp; loki required</h2> <p><a href="https://www.youtube.com/watch?v=kR5ay4lX0OM" target="_blank" rel="noopener noreferrer">https://www.youtube.com/watch?v=kR5ay4lX0OM</a></p> <h2 id="required-nginx-json-log-format-configuration-described-below">Required nginx json log format configuration described below</h2> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">log_format json_analytics escape=json &#39;{&#39; &#39;&#34;msec&#34;: &#34;$msec&#34;, &#39; # request unixtime in seconds with a milliseconds resolution &#39;&#34;connection&#34;: &#34;$connection&#34;, &#39; # connection serial number &#39;&#34;connection_requests&#34;: &#34;$connection_requests&#34;, &#39; # number of requests made in connection &#39;&#34;pid&#34;: &#34;$pid&#34;, &#39; # process pid &#39;&#34;request_id&#34;: &#34;$request_id&#34;, &#39; # the unique request id &#39;&#34;request_length&#34;: &#34;$request_length&#34;, &#39; # request length (including headers and body) &#39;&#34;remote_addr&#34;: &#34;$remote_addr&#34;, &#39; # client IP &#39;&#34;remote_user&#34;: &#34;$remote_user&#34;, &#39; # client HTTP username &#39;&#34;remote_port&#34;: &#34;$remote_port&#34;, &#39; # client port &#39;&#34;time_local&#34;: &#34;$time_local&#34;, &#39; &#39;&#34;time_iso8601&#34;: &#34;$time_iso8601&#34;, &#39; # local time in the ISO 8601 standard format &#39;&#34;request&#34;: &#34;$request&#34;, &#39; # full path no arguments if the request &#39;&#34;request_uri&#34;: &#34;$request_uri&#34;, &#39; # full path and arguments if the request &#39;&#34;args&#34;: &#34;$args&#34;, &#39; # args &#39;&#34;status&#34;: &#34;$status&#34;, &#39; # response status code &#39;&#34;body_bytes_sent&#34;: &#34;$body_bytes_sent&#34;, &#39; # the number of body bytes exclude headers sent to a client &#39;&#34;bytes_sent&#34;: &#34;$bytes_sent&#34;, &#39; # the number of bytes sent to a client &#39;&#34;http_referer&#34;: &#34;$http_referer&#34;, &#39; # HTTP referer &#39;&#34;http_user_agent&#34;: &#34;$http_user_agent&#34;, &#39; # user agent &#39;&#34;http_x_forwarded_for&#34;: &#34;$http_x_forwarded_for&#34;, &#39; # http_x_forwarded_for &#39;&#34;http_host&#34;: &#34;$http_host&#34;, &#39; # the request Host: header &#39;&#34;server_name&#34;: &#34;$server_name&#34;, &#39; # the name of the vhost serving the request &#39;&#34;request_time&#34;: &#34;$request_time&#34;, &#39; # request processing time in seconds with msec resolution &#39;&#34;upstream&#34;: &#34;$upstream_addr&#34;, &#39; # upstream backend server for proxied requests &#39;&#34;upstream_connect_time&#34;: &#34;$upstream_connect_time&#34;, &#39; # upstream handshake time incl. TLS &#39;&#34;upstream_header_time&#34;: &#34;$upstream_header_time&#34;, &#39; # time spent receiving upstream headers &#39;&#34;upstream_response_time&#34;: &#34;$upstream_response_time&#34;, &#39; # time spend receiving upstream body &#39;&#34;upstream_response_length&#34;: &#34;$upstream_response_length&#34;, &#39; # upstream response length &#39;&#34;upstream_cache_status&#34;: &#34;$upstream_cache_status&#34;, &#39; # cache HIT/MISS where applicable &#39;&#34;ssl_protocol&#34;: &#34;$ssl_protocol&#34;, &#39; # TLS protocol &#39;&#34;ssl_cipher&#34;: &#34;$ssl_cipher&#34;, &#39; # TLS cipher &#39;&#34;scheme&#34;: &#34;$scheme&#34;, &#39; # http or https &#39;&#34;request_method&#34;: &#34;$request_method&#34;, &#39; # request method &#39;&#34;server_protocol&#34;: &#34;$server_protocol&#34;, &#39; # request protocol, like HTTP/1.1 or HTTP/2.0 &#39;&#34;pipe&#34;: &#34;$pipe&#34;, &#39; # &#34;p&#34; if request was pipelined, &#34;.&#34; otherwise &#39;&#34;gzip_ratio&#34;: &#34;$gzip_ratio&#34;, &#39; &#39;&#34;http_cf_ray&#34;: &#34;$http_cf_ray&#34;,&#39; &#39;&#34;geoip_country_code&#34;: &#34;$geoip_country_code&#34;&#39; &#39;}&#39;; access_log /var/log/nginx/access.log json_analytics;</code></pre> </div> </div> <p>Adding log_format to sites-available on Nginx :</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">server { .... access_log /var/log/nginx/website_access.log json_analytics;</code></pre> </div> </div> <p>For IP to country mapping, also enable the Geo_IP module:</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">geoip_country /etc/nginx/GeoIP.dat; geoip_city /etc/nginx/GeoIPCity.dat;</code></pre> </div> </div> <p>Option if you are using Cloudflare IP Geo location you can change the log_format :</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">&#39;&#34;geoip_country_code&#34;: &#34;$http_cf_ipcountry&#34;&#39;</code></pre> </div> </div> <p>Cloudflare IP Geo location : <a href="https://support.cloudflare.com/hc/en-us/articles/200168236-Configuring-Cloudflare-IP-Geolocation" target="_blank" rel="noopener noreferrer">https://support.cloudflare.com/hc/en-us/articles/200168236-Configuring-Cloudflare-IP-Geolocation</a></p> <p>Promtail scrapes the log files best when you mount the log volume in the docker container.</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">$ sudo mv promtail-config.yaml /mnt/config/ $ docker create --name promtail --restart unless-stopped -v /mnt/config:/mnt/config -v /var/log:/var/log grafana/promtail:2.1.0 -config.file=/mnt/config/promtail-config.yaml $ docker start promtail</code></pre> </div> </div> <p>Promtail-config.yml file details.</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">console</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-console">server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://&lt;ip address&gt;:3100/loki/api/v1/push scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: varlogs host: nginx01 agent: promtail __path__: /var/log/*log - job_name: nginx static_configs: - targets: - localhost labels: job: nginx host: nginx01 agent: promtail __path__: /var/log/nginx/*log</code></pre> </div> </div>