Loki NGINX Service Mesh - JSON version

<p>Start using Grafana Loki even faster in Grafana Cloud (free plan includes 50GB of Loki logs): <a href="/signup/cloud/connect-account?pg=lokidash">https://grafana.com/signup/cloud/connect-account</a></p> <p>This dashboard is designed for Grafana 8.0 and higher.</p> <p>Required NGINX json log format configuration below.</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">cfg</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-cfg">log_format json_analytics escape=json &#39;{&#39; &#39;&#34;msec&#34;: &#34;$msec&#34;, &#39; # request unixtime in seconds with a milliseconds resolution &#39;&#34;connection&#34;: &#34;$connection&#34;, &#39; # connection serial number &#39;&#34;connection_requests&#34;: &#34;$connection_requests&#34;, &#39; # number of requests made in connection &#39;&#34;pid&#34;: &#34;$pid&#34;, &#39; # process pid &#39;&#34;request_id&#34;: &#34;$request_id&#34;, &#39; # the unique request id &#39;&#34;request_length&#34;: &#34;$request_length&#34;, &#39; # request length (including headers and body) &#39;&#34;remote_addr&#34;: &#34;$remote_addr&#34;, &#39; # client IP &#39;&#34;remote_user&#34;: &#34;$remote_user&#34;, &#39; # client HTTP username &#39;&#34;remote_port&#34;: &#34;$remote_port&#34;, &#39; # client port &#39;&#34;time_local&#34;: &#34;$time_local&#34;, &#39; &#39;&#34;time_iso8601&#34;: &#34;$time_iso8601&#34;, &#39; # local time in the ISO 8601 standard format &#39;&#34;request&#34;: &#34;$request&#34;, &#39; # full path no arguments if the request &#39;&#34;request_uri&#34;: &#34;$request_uri&#34;, &#39; # full path and arguments if the request &#39;&#34;args&#34;: &#34;$args&#34;, &#39; # args &#39;&#34;status&#34;: &#34;$status&#34;, &#39; # response status code &#39;&#34;body_bytes_sent&#34;: &#34;$body_bytes_sent&#34;, &#39; # the number of body bytes exclude headers sent to a client &#39;&#34;bytes_sent&#34;: &#34;$bytes_sent&#34;, &#39; # the number of bytes sent to a client &#39;&#34;http_referer&#34;: &#34;$http_referer&#34;, &#39; # HTTP referer &#39;&#34;http_user_agent&#34;: &#34;$http_user_agent&#34;, &#39; # user agent &#39;&#34;http_x_forwarded_for&#34;: &#34;$http_x_forwarded_for&#34;, &#39; # http_x_forwarded_for &#39;&#34;http_host&#34;: &#34;$http_host&#34;, &#39; # the request Host: header &#39;&#34;server_name&#34;: &#34;$server_name&#34;, &#39; # the name of the vhost serving the request &#39;&#34;request_time&#34;: &#34;$request_time&#34;, &#39; # request processing time in seconds with msec resolution &#39;&#34;upstream&#34;: &#34;$upstream_addr&#34;, &#39; # upstream backend server for proxied requests &#39;&#34;upstream_connect_time&#34;: &#34;$upstream_connect_time&#34;, &#39; # upstream handshake time incl. TLS &#39;&#34;upstream_header_time&#34;: &#34;$upstream_header_time&#34;, &#39; # time spent receiving upstream headers &#39;&#34;upstream_response_time&#34;: &#34;$upstream_response_time&#34;, &#39; # time spend receiving upstream body &#39;&#34;upstream_response_length&#34;: &#34;$upstream_response_length&#34;, &#39; # upstream response length &#39;&#34;upstream_cache_status&#34;: &#34;$upstream_cache_status&#34;, &#39; # cache HIT/MISS where applicable &#39;&#34;ssl_protocol&#34;: &#34;$ssl_protocol&#34;, &#39; # TLS protocol &#39;&#34;ssl_cipher&#34;: &#34;$ssl_cipher&#34;, &#39; # TLS cipher &#39;&#34;scheme&#34;: &#34;$scheme&#34;, &#39; # http or https &#39;&#34;request_method&#34;: &#34;$request_method&#34;, &#39; # request method &#39;&#34;server_protocol&#34;: &#34;$server_protocol&#34;, &#39; # request protocol, like HTTP/1.1 or HTTP/2.0 &#39;&#34;pipe&#34;: &#34;$pipe&#34;, &#39; # &#34;p&#34; if request was pipelined, &#34;.&#34; otherwise &#39;&#34;gzip_ratio&#34;: &#34;$gzip_ratio&#34;, &#39; &#39;&#34;http_cf_ray&#34;: &#34;$http_cf_ray&#34;,&#39; &#39;&#34;geoip_country_code&#34;: &#34;$geoip_country_code&#34;&#39; &#39;}&#39;; access_log /var/log/nginx/json_access.log json_analytics;</code></pre> </div> </div> <p>For IP to country mapping, also enable the Geo_IP module:</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">cfg</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-cfg">geoip_country /etc/nginx/GeoIP.dat; geoip_city /etc/nginx/GeoIPCity.dat;</code></pre> </div> </div> <p>My promtail configuration:</p> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">cfg</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-cfg">server: http_listen_port: 0 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: https://USER:PASSWORD@logs-prod-us-central1.grafana.net/api/prom/push scrape_configs: - job_name: system pipeline_stages: - replace: expression: &#39;(?:[0-9]{1,3}\.){3}([0-9]{1,3})&#39; replace: &#39;***&#39; static_configs: - targets: - localhost labels: job: nginx_access_log host: appfelstrudel agent: promtail __path__: /var/log/nginx/*access.log</code></pre> </div> </div>