Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana documentation Data sources Loki

Using Loki in Grafana

BETA: Querying Loki data requires Grafana’s Explore section. Grafana v6.x comes with Explore enabled by default. In Grafana v5.3.x and v5.4.x. you need to enable Explore manually. Viewing Loki data in dashboard panels is supported in Grafana v6.4+.

Grafana ships with built-in support for Loki, Grafana’s log aggregation system. Just add it as a data source and you are ready to query your log data in Explore.

Adding the data source

1. Open Grafana and make sure you are logged in.
2. In the side menu under the Configuration link you should find a link named Data Sources.
3. Click the Add data source button at the top.
4. Select Loki from the list of data sources.

Note: If you’re not seeing the Data Sources link in your side menu it means that your current user does not have the Admin role for the current organization.

Derived fields

The Derived Fields configuration allows you to:

• Add fields parsed from the log message.
• Add a link that uses the value of the field.

You can use this functionality to link to your tracing backend directly from your logs, or link to a user profile page if a userId is present in the log line. These links appear in the log details.

• Name - Shown in the log details as a label.
• Regex - A Regex pattern that runs on the log message and captures part of it as the value of the new field. Can only contain a single capture group.
• URL/query - If the link is external, then enter the full link URL. If the link is internal link, then this input serves as query for the target data source. In both cases, you can interpolate the value from the field with ${__value.raw } macro.
• Internal link - Select if the link is internal or external. In case of internal link, a data source selector allows you to select the target data source. Only tracing data sources are supported.

You can use a debug section to see what your fields extract and how the URL is interpolated. Click Show example log message to show the text area where you can enter a log message.

The new field with the link shown in log details:

Querying Logs

Querying and displaying log data from Loki is available via Explore, and with the logs panel in dashboards. Select the Loki data source, and then enter a LogQL query to display your logs.

Log Queries

A log query consists of two parts: log stream selector, and a search expression. For performance reasons you need to start by choosing a log stream by selecting a log label.

The Logs Explorer (the Log labels button) next to the query field shows a list of labels of available log streams. An alternative way to write a query is to use the query field’s autocomplete - you start by typing a left curly brace { and the autocomplete menu will suggest a list of labels. Press the Enter key to execute the query.

Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.

Log Stream Selector

For the label part of the query expression, wrap it in curly braces {} and then use the key value syntax for selecting labels. Multiple label expressions are separated by a comma:

{app="mysql",name="mysql-backup"}

The following label matching operators are currently supported:

• = exactly equal.
• != not equal.
• =~ regex-match.
• !~ do not regex-match.

Examples:

• {name=~"mysql.+"}
• {name!~"mysql.+"}

The same rules that apply for Prometheus Label Selectors apply for Loki Log Stream Selectors.

Another way to add a label selector is in the table section. Click Filter beside a label to add the label to the query expression. This even works for multiple queries and will add the label selector to each query.

Search Expression

After writing the Log Stream Selector, you can filter the results further by writing a search expression. The search expression can be just text or a regex expression.

Example queries:

• {job="mysql"} |= "error"
• {name="kafka"} |~ "tsdb-ops.*io:2003"
• {instance=~"kafka-[23]",name="kafka"} != "kafka.server:type=ReplicaManager"

Filter operators can be chained and will sequentially filter down the expression. The resulting log lines will satisfy every filter.

Example

{job="mysql"} |= "error" != "timeout"

The following filter types are currently supported:

• |= line contains string.
• != line doesn’t contain string.
• |~ line matches regular expression.
• !~ line does not match regular expression.

Note: For more details about LogQL, Loki’s query language, refer to the Loki LogQL

Live tailing

Loki supports Live tailing which displays logs in real-time. This feature is supported in Explore.

Note that Live Tailing relies on two Websocket connections: one between the browser and the Grafana server, and another between the Grafana server and the Loki server. If you run any reverse proxies, please configure them accordingly. The following example for Apache2 can be used for proxying between the browser and the Grafana server:

Copy

ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"

The following example shows basic NGINX proxy configuration. It assumes that the Grafana server is available at http://localhost:3000/, Loki server is running locally without proxy, and your external site uses HTTPS. If you also host Loki behind NGINX proxy, then you might want to repeat the following configuration for Loki as well.

In the http section of NGINX configuration, add the following map definition:

Copy

  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

In your server section, add the following configuration:

Copy

  location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
      proxy_pass          http://localhost:3000$request_uri;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
      proxy_set_header    Connection        $connection_upgrade;
      proxy_set_header    Upgrade           $http_upgrade;
  }

  location / {
      proxy_pass          http://localhost:3000/;
      proxy_set_header    Host              $host;
      proxy_set_header    X-Real-IP         $remote_addr;
      proxy_set_header    X-Forwarded-for   $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto "https";
  }

Note: This feature is only available in Grafana v6.3+.

Log Context

When using a search expression as detailed above, you now have the ability to retrieve the context surrounding your filtered results. By clicking the Show Context link on the filtered rows, you’ll be able to investigate the log messages that came before and after the log message you’re interested in.

Note: This feature is only available in Grafana v6.3+

Templating

Instead of hard-coding things like server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as drop-down select boxes at the top of the dashboard. These drop-down boxes make it easy to change the data being displayed in your dashboard.

Check out the Templating documentation for an introduction to the templating feature and the different types of template variables.

Annotations

You can use any non-metric Loki query as a source for annotations. Log content will be used as annotation text and your log stream labels as tags, so there is no need for additional mapping.

Note: Annotations for Loki are only available in Grafana v6.4+

Configure the data source with provisioning

You can set up the data source via config files with Grafana’s provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page

Here is an example:

yaml Copy

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      maxLines: 1000

Here’s another with basic auth and derived field. Keep in mind that $ character needs to be escaped in YAML values as it is used to interpolate environment variables:

yaml Copy

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://localhost:3100
    basicAuth: true
    basicAuthUser: my_user
    basicAuthPassword: test_password
    jsonData:
      maxLines: 1000
      derivedFields:
        # Field with internal link pointing to data source in Grafana.
        # Right now, Grafana supports only Jaeger and Zipkin data sources as link targets.
        # datasourceUid value can be anything, but it should be unique across all defined data source uids.
        - datasourceUid: my_jaeger_uid
          matcherRegex: "traceID=(\\w+)"
          name: TraceID
          # url will be interpreted as query for the datasource
          url: "$${__value.raw}"

        # Field with external link.
        - matcherRegex: "traceID=(\\w+)"
          name: TraceID
          url: "http://localhost:16686/trace/$${__value.raw}"

Here’s an example of a Jaeger data source corresponding to the above example. Note that the Jaeger uid value does match the Loki datasourceUid value.

Copy

datasources:
    - name: Jaeger
      type: jaeger
      url: http://jaeger-tracing-query:16686/
      access: proxy
      # UID should match the datasourceUid in dervidedFields.
      uid: my_jaeger_uid