Query Loki on Grafana Labs

Query best practices

Thu, 09 Apr 2026 02:28:18 +0000

Query best practices

The way you write queries in Loki affects how quickly you get results returned from those queries. Understanding the way Loki parses queries can help you write queries that are efficient and performant.

Tip
Before you start optimizing queries, read the labels best practices page to understand what makes a good label. Choosing the right labels is the first step towards writing efficient queries.

Loki evaluates a LogQL query from left to right, in the order that it is written. To get the best possible query performance, eliminate as many potential results as you can earlier in the query and then continue to progressively narrow your search as you continue writing the query. This page describes the recommended order for writing queries that efficiently filter out unwanted results.

Narrow down your time range first

Reduce the number of logs Loki needs to look through by specifying a period of time that you’d like to search through. Loki creates one index file per day, so queries that span over multiple days fetches multiple index files. The fewer files Loki has to search, the faster the query results are returned.

Time ranges are typically not part of the query, but you can set a time range through your visualization tool or through the Loki API.

In Grafana

If you’re using Loki with Grafana, you can use the dropdown menu on the upper right hand corner of a dashboard to select a time range, either relative (last X hours) or absolute (a specific date and time).

Through Loki API

If you’re querying Loki through the Loki API, you can use the query_range endpoint to add start and end timestamps for your query as parameters to the HTTP call rather than as part of the query itself.

http://<loki-instance>/loki/api/v1/query_range?query={job="app"}&start=1633017600000000000&end=1633104000000000000

Use precise label selectors

Next, write your label selectors. Identify the most specific label you can use within the log line and search based on that first. For example, if the logs contain the labels namespace and app_name and the latter is a smaller subset of data, start your query by selecting based on app_name:

{app_name="carnivorousgreenhouse"}

Using the most specific label selector has the added benefit of reducing the length of your query. Since app_name is more specific than namespace, you don’t need to add a selector for namespace. Adding more general label selectors has no further effect on the query.

Use simple line filter expressions over regular expressions

When using line filter expressions, prefer the simpler filter operators such as:

|= (contains string) and
!= (does not contain string) over the regular expression filter operators:
|~ (matches the regular expression)
!~ (does not match the regular expression)

Loki evaluates the first two filter expressions faster than it can evaluate regular expressions, so always try to rewrite your query in terms of whether a log line contains or does not contain a certain string. Use regular expressions only as a last resort.

Line filter expressions are more efficient than parser expressions.

Avoid using complex text parsers

Use parser expressions only after line filter expressions. Parser expressions are ways to look through the log line and extract labels in different formats, which can be useful but are also more intensive for Loki to do than line filter expressions. Using them after line filter expressions means that Loki only needs to evaluate parser expressions for log lines that match the line filter expression, reducing the amount of logs that Loki needs to search through.

Parser expressions include JSON, logfmt, pattern, regexp, and unpack parsers.

Use recording rules

Some queries are sufficiently complex, or some datasets sufficiently large, that there is a limit as to how much query performance can be optimized. If you’re following the tips on this page and are still experiencing slow query times, consider creating a recording rule for them. A recording rule runs a query at a predetermined time and also precomputes the results of that query, saving those results for faster retrieval later.

Further resources

Simple LogQL simulator

Thu, 09 Apr 2026 02:28:18 +0000

Simple LogQL simulator

The LogQL simulator is an online tool that you can use to experiment with writing simple LogQL queries and seeing the results, without needing to run an instance of Loki.

A set of example log lines are included for each of the primary log parsers supported by Loki:

Logfmt
JSON
Unstructured text, which can be parsed with the Loki pattern or regex parsers

The log stream selector {job="analyze"} is shown as an example, and it remains fixed for all possible example queries in the simulator. A log stream is a set of logs which share the same labels. In LogQL, you use a log stream selector to determine which log streams to include in a query’s results.

Note
This is a very limited simulator, primarily for evaluating filters and parsers. If you want to practice writing more complex queries, such as metric queries, you can use the Explore feature in Grafana.

To use the LogQL simulator:

Select a log line format using the radio buttons.
You can use the provided example log lines, or copy and paste your own log lines into the example log lines box.
Use the provided example LogQL query, or enter your own query. The log stream selector remains fixed for all possible example queries. There are additional sample queries at the end of this topic.
Click the Run query button to run the entered query against the example log lines.

The results output simulates how Loki would return results for your query. You can also click each line in the results pane to expand the details, which give an explanation for why the log line is or is not included in the query result set.

Log line format: logfmt JSON Unstructured text

{job="analyze"}

Query:

{job="analyze"}

Additional sample queries

These are some additional sample queries that you can use in the LogQL simulator.

Logfmt

| logfmt | level = "debug"

Parses logfmt-formatted logs and returns only log lines where the “level” field is equal to “debug”.

| logfmt | msg="server listening on addresses"

Parses logfmt-formatted logs and returns only log lines with the message “server listening on address.”

JSON

| json | level="INFO" | file="SpringApplication.java" | line_format `{{.class}}`

Parses JSON-formatted logs, filtering for lines where the ’level’ field is “INFO” and the ‘file field is “SpringApplication.java”, then formats the line to return only the ‘class’ field.

|~ `(T|t)omcat`

Performs a regular expression filter for the string ’tomcat’ or ‘Tomcat’, without using a parser.

Unstructured text

| pattern "<_> - <_> <_> \"<method> <url> <protocol>\" <status> <_> <_> \"<_>\" <_>" | method="GET"

Parses unstructured logs with the pattern parser, filtering for lines where the HTTP method is “GET”.

| pattern "<_> - <user> <_> \"<method> <url> <protocol>\" <status> <_> <_> \"<_>\" <_>" | user=~"kling.*"

Parses unstructured logs with the pattern parser, extracting the ‘user’ field, and filtering for lines where the user field starts with “kling”.

Log queries

Thu, 09 Apr 2026 02:28:18 +0000

Log queries

All LogQL queries contain a log stream selector.

Optionally, the log stream selector can be followed by a log pipeline. A log pipeline is a set of stage expressions that are chained together and applied to the selected log streams. Each expression can filter out, parse, or mutate log lines and their respective labels.

The following example shows a full log query in action:

{container="query-frontend",namespace="loki-dev"} |= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500

The query is composed of:

a log stream selector {container="query-frontend",namespace="loki-dev"} which targets the query-frontend container in the loki-dev namespace.
a log pipeline |= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500 which will filter out log that contains the word metrics.go, then parses each log line to extract more labels and filter with them.

To avoid escaping special characters you can use the `(backtick) instead of " when quoting strings. For example `\w+` is the same as "\\w+". This is specially useful when writing a regular expression which contains multiple backslashes that require escaping.

Log stream selector

The stream selector determines which log streams to include in a query’s results. A log stream is a unique source of log content, such as a file. A more granular log stream selector then reduces the number of searched streams to a manageable volume. This means that the labels passed to the log stream selector will affect the relative performance of the query’s execution.

The log stream selector is specified by one or more comma-separated key-value pairs. Each key is a log label and each value is that label’s value. Curly braces ({ and }) delimit the stream selector.

Consider this stream selector:

{app="mysql",name="mysql-backup"}

All log streams that have both a label of app whose value is mysql and a label of name whose value is mysql-backup will be included in the query results. A stream may contain other pairs of labels and values, but only the specified pairs within the stream selector are used to determine which streams will be included within the query results.

The same rules that apply for Prometheus Label Selectors apply for Grafana Loki log stream selectors.

The = operator after the label name is a label matching operator. The following label matching operators are supported:

=: exactly equal
!=: not equal
=~: regex matches
!~: regex does not match

Regex log stream examples:

{name =~ "mysql.+"}
{name !~ "mysql.+"}
{name !~ `mysql-\d+`}

Note: Unlike the line filter regex expressions, the =~ and !~ regex operators are fully anchored. This means that the regex expression must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term\n. Alternatively, you can use the \s (match whitespaces, including newline) in combination with \S (match not whitespace characters) to match all characters, including newlines. Refer to Google’s RE2 syntax for more information.

Regex log stream newlines:

{name =~ ".*mysql.*"}: does not match log label values with newline character
{name =~ "(?s).*mysql.*}: match log label values with newline character
{name =~ "[\S\s]*mysql[\S\s]*}: match log label values with newline character

Log pipeline

A log pipeline can be appended to a log stream selector to further process and filter log streams. It is composed of a set of expressions. Each expression is executed in left to right sequence for each log line. If an expression filters out a log line, the pipeline will stop processing the current log line and start processing the next log line.

Some expressions can mutate the log content and respective labels, which will then be available for further filtering and processing in subsequent expressions. An example that mutates is the expression

| line_format "{{.status_code}}"

Log pipeline expressions fall into one of four categories:

Filtering expressions: line filter expressions and label filter expressions
Parsing expressions
Formatting expressions: line format expressions and label format expressions
Labels expressions: drop labels expression and keep labels expression

Line filter expression

The line filter expression does a distributed grep over the aggregated logs from the matching log streams. It searches the contents of the log line, discarding those lines that do not match the case-sensitive expression.

Each line filter expression has a filter operator followed by text or a regular expression. These filter operators are supported:

|=: Log line contains string
!=: Log line does not contain string
|~: Log line contains a match to the regular expression
!~: Log line does not contain a match to the regular expression

Note: Unlike the label matcher regex operators, the |~ and !~ regex operators are not fully anchored. This means that the . regex character matches all characters, including newlines.

Line filter expression examples:

Keep log lines that have the substring “error”:
```
|= "error"
```
A complete query using this example:
```
{job="mysql"} |= "error"
```
Discard log lines that have the substring “kafka.server:type=ReplicaManager”:
```
!= "kafka.server:type=ReplicaManager"
```
A complete query using this example:
```
{instance=~"kafka-[23]",name="kafka"} != "kafka.server:type=ReplicaManager"
```
Keep log lines that contain a substring that starts with tsdb-ops and ends with io:2003. A complete query with a regular expression:
```
{name="kafka"} |~ "tsdb-ops.*io:2003"
```
Keep log lines that contain a substring that starts with error=, and is followed by 1 or more word characters. A complete query with a regular expression:
```
{name="cassandra"} |~  `error=\w+`
```

Filter operators can be chained. Filters are applied sequentially. Query results will have satisfied every filter. This complete query example will give results that include the string error, and do not include the string timeout.

{job="mysql"} |= "error" != "timeout"

When using |~ and !~, Go (as in Golang) RE2 syntax regex may be used. The matching is case-sensitive by default. Switch to case-insensitive matching by prefixing the regular expression with (?i).

While line filter expressions could be placed anywhere within a log pipeline, it is almost always better to have them at the beginning. Placing them at the beginning improves the performance of the query, as it only does further processing when a line matches. For example, while the results will be the same, the query specified with

{job="mysql"} |= "error" | json | line_format "{{.err}}"

will always run faster than

{job="mysql"} | json | line_format "{{.message}}" |= "error"

Line filter expressions are the fastest way to filter logs once the log stream selectors have been applied.

Line filter expressions have support matching IP addresses. See Matching IP addresses for details.

Removing color codes

Line filter expressions support stripping ANSI sequences (color codes) from the line:

{job="example"} | decolorize

Label filter expression

Label filter expression allows filtering log line using their original and extracted labels. It can contain multiple predicates.

A predicate contains a label identifier, an operation and a value to compare the label with.

For example with cluster="namespace" the cluster is the label identifier, the operation is = and the value is “namespace”. The label identifier is always on the left side of the operation.

We support multiple value types which are automatically inferred from the query input.

String is double quoted or backticked such as "200" or `us-central1`.
Duration is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. The value of the label identifier used for comparison must be a string with a unit suffix to be parsed correctly, such as “0.10ms” or “1h30m”. Optionally, label_format can be used to modify the value and append the unit before making the comparison.
Number are floating-point number (64bits), such as250, 89.923.
Bytes is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “42MB”, “1.5KiB” or “20B”. Valid bytes units are “B”, “kB”, “MB”, “GB”, “TB”, “KB”, “KiB”, “MiB”, “GiB”, “TiB”.

String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).

The string type is the only one that can filter out a log line with a label __error__.

Using Duration, Number and Bytes will convert the label value prior to comparison and support the following comparators:

== or = for equality.
!= for inequality.
> and >= for greater than and greater than or equal.
< and <= for lesser than and lesser than or equal.

For instance, logfmt | duration > 1m and bytes_consumed > 20MB

If the conversion of the label value fails, the log line is not filtered and an __error__ label is added. To filters those errors see the pipeline errors section.

You can chain multiple predicates using and and or which respectively express the and and or binary operations. and can be equivalently expressed by a comma, a space or another pipe. Label filters can be place anywhere in a log pipeline.

This means that all the following expressions are equivalent:

| duration >= 20ms or size == 20KB and method!~"2.."
| duration >= 20ms or size == 20KB | method!~"2.."
| duration >= 20ms or size == 20KB , method!~"2.."
| duration >= 20ms or size == 20KB  method!~"2.."

The precedence for evaluation of multiple predicates is left to right. You can wrap predicates with parenthesis to force a different precedence.

These examples are equivalent:

| duration >= 20ms or method="GET" and size <= 20KB
| ((duration >= 20ms or method="GET") and size <= 20KB)

To evaluate the logical and first, use parenthesis, as in this example:

| duration >= 20ms or (method="GET" and size <= 20KB)

Label filter expressions are the only expression allowed after the unwrap expression. This is mainly to allow filtering errors from the metric extraction.

Label filter expressions have support matching IP addresses. See Matching IP addresses for details.

Parser expression

Parser expression can parse and extract labels from the log content. Those extracted labels can then be used for filtering using label filter expressions or for metric aggregations.

Extracted label keys are automatically sanitized by all parsers, to follow Prometheus metric name convention.(They can only contain ASCII letters and digits, as well as underscores and colons. They cannot start with a digit.)

For instance, the pipeline | json will produce the following mapping:

{ "a.b": {c: "d"}, e: "f" }

{a_b_c="d", e="f"}

In case of errors, for instance if the line is not in the expected format, the log line won’t be filtered but instead will get a new __error__ label added.

If an extracted label key name already exists in the original log stream, the extracted label key will be suffixed with the _extracted keyword to make the distinction between the two labels. You can forcefully override the original label using a label formatter expression. However, if an extracted key appears twice, only the first label value will be kept.

Loki supports JSON, logfmt, pattern, regexp and unpack parsers.

It’s easier to use the predefined parsers json and logfmt when you can. If you can’t, the pattern and regexp parsers can be used for log lines with an unusual structure. The pattern parser is easier and faster to write; it also outperforms the regexp parser. Multiple parsers can be used by a single log pipeline. This is useful for parsing complex logs. There are examples in Multiple parsers.

JSON

The json parser operates in two modes:

without parameters:

Adding | json to your pipeline will extract all json properties as labels if the log line is a valid json document. Nested properties are flattened into label keys using the _ separator.

Note: Arrays are skipped.

For example the json parsers will extract from the following document:

{
    "protocol": "HTTP/2.0",
    "servers": ["129.0.1.1","10.2.1.3"],
    "request": {
        "time": "6.032",
        "method": "GET",
        "host": "foo.grafana.net",
        "size": "55",
        "headers": {
          "Accept": "*/*",
          "User-Agent": "curl/7.68.0"
        }
    },
    "response": {
        "status": 401,
        "size": "228",
        "latency_seconds": "6.031"
    }
}

The following list of labels:

"protocol" => "HTTP/2.0"
"request_time" => "6.032"
"request_method" => "GET"
"request_host" => "foo.grafana.net"
"request_size" => "55"
"request_headers_Accept" => "*/*"
"request_headers_User_Agent" => "curl/7.68.0"
"response_status" => "401"
"response_size" => "228"
"response_latency_seconds" => "6.031"

with parameters:

Using | json label="expression", another="expression" in your pipeline will extract only the specified json fields to labels. You can specify one or more expressions in this way, the same as label_format; all expressions must be quoted.

Currently, we only support field access (my.field, my["field"]) and array access (list[0]), and any combination of these in any level of nesting (my.list[0]["field"]).

For example, | json first_server="servers[0]", ua="request.headers[\"User-Agent\"] will extract from the following document:
JSON
```
{
    "protocol": "HTTP/2.0",
    "servers": ["129.0.1.1","10.2.1.3"],
    "request": {
        "time": "6.032",
        "method": "GET",
        "host": "foo.grafana.net",
        "size": "55",
        "headers": {
          "Accept": "*/*",
          "User-Agent": "curl/7.68.0"
        }
    },
    "response": {
        "status": 401,
        "size": "228",
        "latency_seconds": "6.031"
    }
}
```
The following list of labels:
kv
```
"first_server" => "129.0.1.1"
"ua" => "curl/7.68.0"
```
If an array or an object returned by an expression, it will be assigned to the label in json format.

For example, | json server_list="servers", headers="request.headers" will extract:
kv
```
"server_list" => `["129.0.1.1","10.2.1.3"]`
"headers" => `{"Accept": "*/*", "User-Agent": "curl/7.68.0"}`
```
If the label to be extracted is same as the original JSON field, expression can be written as just | json <label>

For example, to extract servers fields as label, expression can be written as following

| json servers will extract:
kv
```
"servers" => `["129.0.1.1","10.2.1.3"]`
```
Note that | json servers is same as | json servers="servers"

logfmt

The logfmt parser can operate in two modes:

without parameters:

The logfmt parser can be added using | logfmt and will extract all keys and values from the logfmt formatted log line.

For example the following log line:
logfmt
```
at=info method=GET path=/ host=grafana.net fwd="124.133.124.161" service=8ms status=200
```
will result in having the following labels extracted:
kv
```
"at" => "info"
"method" => "GET"
"path" => "/"
"host" => "grafana.net"
"fwd" => "124.133.124.161"
"service" => "8ms"
"status" => "200"
```
with parameters:

Similar to JSON, using | logfmt label="expression", another="expression" in the pipeline will result in extracting only the fields specified by the labels.

For example, | logfmt host, fwd_ip="fwd" will extract the labels host and fwd from the following log line:
logfmt
```
at=info method=GET path=/ host=grafana.net fwd="124.133.124.161" service=8ms status=200
```
And rename fwd to fwd_ip:
kv
```
"host" => "grafana.net"
"fwd_ip" => "124.133.124.161"
```

The logfmt parser also supports the following flags:

--strict to enable strict parsing

With strict parsing enabled, the logfmt parser immediately stops scanning the log line and returns early with an error when it encounters any poorly formatted key/value pair.
```
// accepted key/value pairs
key=value key="value in double quotes"

// invalid key/value pairs
=value // no key
foo=bar=buzz
fo"o=bar
```
Without the --strict flag the parser skips invalid key/value pairs and continues parsing the rest of the log line. Non-strict mode offers the flexibility to parse semi-structed log lines, though note that this is only best-effort.
--keep-empty to retain standalone keys with empty value

With --keep-empty flag set, the logfmt parser retains standalone keys(keys without a value) as labels with value set to empty string. If the standalone key is explicitly requested using label extraction parameters, there is no need to add this flag.

Note: flags if any should appear right after logfmt and before label extraction parameters

| logfmt --strict
| logfmt --strict host, fwd_ip="fwd"
| logfmt --keep-empty --strict host

Pattern

The pattern parser allows the explicit extraction of fields from log lines by defining a pattern expression (| pattern "<pattern-expression>"). The expression matches the structure of a log line.

Consider this NGINX log line.

0.191.12.2 - - [10/Jun/2021:09:14:29 +0000] "GET /api/plugins/versioncheck HTTP/1.1" 200 2 "-" "Go-http-client/2.0" "13.76.247.102, 34.120.177.193" "TLSv1.2" "US" ""

This log line can be parsed with the expression

<ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>

to extract these fields:

"ip" => "0.191.12.2"
"method" => "GET"
"uri" => "/api/plugins/versioncheck"
"status" => "200"
"size" => "2"
"agent" => "Go-http-client/2.0"

A pattern expression is composed of captures and literals.

A capture is a field name delimited by the < and > characters. <example> defines the field name example. An unnamed capture appears as <_>. The unnamed capture skips matched content.

Captures are matched from the line beginning or the previous set of literals, to the line end or the next set of literals. If a capture is not matched, the pattern parser will stop.

Literals can be any sequence of UTF-8 characters, including whitespace characters.

By default, a pattern expression is anchored at the start of the log line. If the expression starts with literals, then the log line must also start with the same set of literals. Use <_> at the beginning of the expression if you don’t want to anchor the expression at the start.

Consider the log line

level=debug ts=2021-06-10T09:24:13.472094048Z caller=logging.go:66 traceID=0568b66ad2d9294c msg="POST /loki/api/v1/push (204) 16.652862ms"

To match msg=", use the expression:

<_> msg="<method> <path> (<status>) <latency>"

A pattern expression is invalid if

It does not contain any named capture.
It contains two consecutive captures not separated by whitespace characters.

Regular expression

Unlike the logfmt and json, which extract implicitly all values and takes no parameters, the regexp parser takes a single parameter | regexp "<re>" which is the regular expression using the Golang RE2 syntax.

The regular expression must contain a least one named sub-match (e.g (?P<name>re)), each sub-match will extract a different label.

For example the parser | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \$(?P<status>\\d+?)\$ (?P<duration>.*)" will extract from the following line:

POST /api/prom/api/v1/query_range (200) 1.5s

those labels:

"method" => "POST"
"path" => "/api/prom/api/v1/query_range"
"status" => "200"
"duration" => "1.5s"

unpack

The unpack parser parses a JSON log line, unpacking all embedded labels from the pack stage. A special property _entry will also be used to replace the original log line.

For example, using | unpack with the log line:

{
  "container": "myapp",
  "pod": "pod-3223f",
  "_entry": "original log message"
}

extracts the container and pod labels; it sets original log message as the new log line.

You can combine the unpack and json parsers (or any other parsers) if the original embedded log line is of a specific format.

Line format expression

The line format expression can rewrite the log line content by using the text/template format. It takes a single string parameter | line_format "{{.label_name}}", which is the template format. All labels are injected variables into the template and are available to use with the {{.label_name}} notation.

For example the following expression:

{container="frontend"} | logfmt | line_format "{{.query}} {{.duration}}"

Will extract and rewrite the log line to only contains the query and the duration of a request.

You can use double quoted string for the template or backticks `{{.label_name}}` to avoid the need to escape special characters.

line_format also supports math functions. Example:

If we have the following labels ip=1.1.1.1, status=200 and duration=3000(ms), we can divide the duration by 1000 to get the value in seconds.

{container="frontend"} | logfmt | line_format "{{.ip}} {{.status}} {{div .duration 1000}}"

The above query will give us the line as 1.1.1.1 200 3

Additionally, you can also access the log line using the __line__ function and the timestamp using the __timestamp__ function. See template functions to learn about available functions in the template format.

Labels format expression

The | label_format expression can rename, modify or add labels. It takes as parameter a comma separated list of equality operations, enabling multiple operations at once.

When both side are label identifiers, for example dst=src, the operation will rename the src label into dst.

The right side can alternatively be a template string (double quoted or backtick), for example dst="{{.status}} {{.query}}", in which case the dst label value is replaced by the result of the text/template evaluation. This is the same template engine as the | line_format expression, which means labels are available as variables and you can use the same list of functions.

In both cases, if the destination label doesn’t exist, then a new one is created.

The renaming form dst=src will drop the src label after remapping it to the dst label. However, the template form will preserve the referenced labels, such that dst="{{.src}}" results in both dst and src having the same value.

A single label name can only appear once per expression. This means | label_format foo=bar,foo="new" is not allowed but you can use two expressions for the desired effect: | label_format foo=bar | label_format foo="new"

Drop Labels expression

Syntax: |drop name, other_name, some_name="some_value"

The | drop expression will drop the given labels in the pipeline. For example, for the query {job="varlogs"}|json|drop level, method="GET", with below log line

{"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{host="grafana.net", path="/", status="200"} {"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

Similarly, this expression can be used to drop __error__ labels as well. For example, for the query {job="varlogs"}|json|drop __error__, with below log line

INFO GET / loki.net 200

the result will be

{} INFO GET / loki.net 200

Example with regex and multiple names

For the query {job="varlogs"}|json|drop level, path, app=~"some-api.*", with below log lines

{"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{host="grafana.net", job="varlogs", method="GET", status="200"} {"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{app="other-service", host="grafana.net", job="varlogs", method="GET", status="200"} {"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

Keep Labels expression

Syntax: |keep name, other_name, some_name="some_value"

The | keep expression will keep only the specified labels in the pipeline and drop all the other labels.

Note
The keep stage will not drop the error or error_details labels added by Loki at query time. To drop these labels, refer to drop stage.

Query examples:

For the query {job="varlogs"}|json|keep level, method="GET", with the following log lines:

{"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{"level": "info", "method": "POST", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{level="info", method="GET"} {"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{level="info"} {"level": "info", "method": "POST", "path": "/", "host": "grafana.net", "status": "200"}

For the query {job="varlogs"}|json|keep level, tenant, app=~"some-api.*", with the following log lines:

{"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{app="some-api-service", level="info"} {"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{level="info"} {"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

Metric queries

Thu, 09 Apr 2026 02:28:18 +0000

Metric queries

Metric queries extend log queries by applying a function to log query results. This powerful feature creates metrics from logs.

Metric queries can be used to calculate the rate of error messages or the top N log sources with the greatest quantity of logs over the last 3 hours.

Combined with parsers, metric queries can also be used to calculate metrics from a sample value within the log line, such as latency or request size. All labels, including extracted ones, will be available for aggregations and generation of new series.

Range Vector aggregation

LogQL shares the range vector concept of Prometheus. In Grafana Loki, the selected range of samples is a range of selected log or label values.

The aggregation is applied over a time duration. Loki defines Time Durations with the same syntax as Prometheus.

Loki supports two types of range vector aggregations: log range aggregations and unwrapped range aggregations.

Log range aggregations

A log range aggregation is a query followed by a duration. A function is applied to aggregate the query over the duration. The duration can be placed after the log stream selector or at end of the log pipeline.

The functions:

rate(log-range): calculates the number of entries per second
count_over_time(log-range): counts the entries for each log stream within the given range.
bytes_rate(log-range): calculates the number of bytes per second for each stream.
bytes_over_time(log-range): counts the amount of bytes used by each log stream for a given range.
absent_over_time(log-range): returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements. (absent_over_time is useful for alerting on when no time series and logs stream exist for label combination for a certain amount of time.)

Examples:

Count all the log lines within the last five minutes for the MySQL job.
logql
```
count_over_time({job="mysql"}[5m])
```
This aggregation includes filters and parsers. It returns the per-second rate of all non-timeout errors within the last minutes per host for the MySQL job and only includes errors whose duration is above ten seconds.
logql
```
sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
```

Offset modifier

The offset modifier allows changing the time offset for individual range vectors in a query.

For example, the following expression counts all the logs within the last ten minutes to five minutes rather than last five minutes for the MySQL job. Note that the offset modifier always needs to follow the range vector selector immediately.

count_over_time({job="mysql"}[5m] offset 5m) // GOOD
count_over_time({job="mysql"}[5m]) offset 5m // INVALID

Unwrapped range aggregations

Unwrapped ranges uses extracted labels as sample values instead of log lines. However to select which label will be used within the aggregation, the log query must end with an unwrap expression and optionally a label filter expression to discard errors.

The unwrap expression is noted | unwrap label_identifier where the label identifier is the label name to use for extracting sample values.

Since label values are string, by default a conversion into a float (64bits) will be attempted, in case of failure the __error__ label is added to the sample. Optionally the label identifier can be wrapped by a conversion function | unwrap <function>(label_identifier), which will attempt to convert the label value from a specific format.

We currently support the functions:

duration_seconds(label_identifier) (or its short equivalent duration) which will convert the label value in seconds from the go duration format (e.g 5m, 24s30ms).
bytes(label_identifier) which will convert the label value to raw bytes applying the bytes unit (e.g. 5 MiB, 3k, 1G).

Supported function for operating over unwrapped ranges are:

rate(unwrapped-range): calculates per second rate of the sum of all values in the specified interval.
rate_counter(unwrapped-range): calculates per second rate of the values in the specified interval and treating them as “counter metric”
sum_over_time(unwrapped-range): the sum of all values in the specified interval.
avg_over_time(unwrapped-range): the average value of all points in the specified interval.
max_over_time(unwrapped-range): the maximum value of all points in the specified interval.
min_over_time(unwrapped-range): the minimum value of all points in the specified interval
first_over_time(unwrapped-range): the first value of all points in the specified interval
last_over_time(unwrapped-range): the last value of all points in the specified interval
stdvar_over_time(unwrapped-range): the population standard variance of the values in the specified interval.
stddev_over_time(unwrapped-range): the population standard deviation of the values in the specified interval.
quantile_over_time(scalar,unwrapped-range): the φ-quantile (0 ≤ φ ≤ 1) of the values in the specified interval.
absent_over_time(unwrapped-range): returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements. (absent_over_time is useful for alerting on when no time series and logs stream exist for label combination for a certain amount of time.)

Except for sum_over_time,absent_over_time, rate and rate_counter, unwrapped range aggregations support grouping.

<aggr-op>([parameter,] <unwrapped-range>) [without|by (<label list>)]

Which can be used to aggregate over distinct labels dimensions by including a without or by clause.

without removes the listed labels from the result vector, while all other labels are preserved the output. by does the opposite and drops labels that are not listed in the by clause, even if their label values are identical between all elements of the vector.

See Unwrap examples for query examples that use the unwrap expression.

Built-in aggregation operators

Like PromQL, LogQL supports a subset of built-in aggregation operators that can be used to aggregate the element of a single vector, resulting in a new vector of fewer elements but with aggregated values:

sum: Calculate sum over labels
avg: Calculate the average over labels
min: Select minimum over labels
max: Select maximum over labels
stddev: Calculate the population standard deviation over labels
stdvar: Calculate the population standard variance over labels
count: Count number of elements in the vector
topk: Select largest k elements by sample value
bottomk: Select smallest k elements by sample value
sort: returns vector elements sorted by their sample values, in ascending order.
sort_desc: Same as sort, but sorts in descending order.

The aggregation operators can either be used to aggregate over all label values or a set of distinct label values by including a without or a by clause:

<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]

parameter is required when using topk and bottomk. topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector.

by and without are only used to group the input vector. The without clause removes the listed labels from the resulting vector, keeping all others. The by clause does the opposite, dropping labels that are not listed in the clause, even if their label values are identical between all elements of the vector.

See vector aggregation examples for query examples that use vector aggregation expressions.

Functions

LogQL supports a set of built-in functions.

vector(s scalar): returns the scalar s as a vector with no labels. This behaves identically to the Prometheus vector() function. vector is mainly used to return a value for a series that would otherwise return nothing; this can be useful when using LogQL to define an alert.

Examples:

Count all the log lines within the last five minutes for the traefik namespace.

sum(count_over_time({namespace="traefik"}[5m])) # will return nothing
  or
vector(0) # will return 0

Probabilistic aggregation

Note
Probabilistic aggregation is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. To use this feature, set limits_config.shard_aggregations:approx_topk in your Loki configuration. To enable this feature in Grafana Cloud, contact Grafana Support.

LogQL’s approx_topk function provides a probabilistic approximation of topk. It is a drop-in replacement for topk that is great for when topk queries time out or hit the maximum series limit. This tends to happen when the list of values that you’re sorting through in order to find the most frequent values is very large. approx_topk is also great in cases where a faster, approximate answer is preferred to a slower, more accurate one.

The function is of the form:

approx_topk(k, <vector expression>)

approx_topk is only supported for instant queries. Grouping is also not supported and should be handled by an inner sum by or sum without even though this might not be the same behavior as topk by.

Under the hood, approx_topk is implemented using sharding. The count-min sketch algorithm and a heap are used to approximate the counts for each shard. The accuracy of the approximation depends on the size of the heap, which is defined by Loki’smax_count_min_sketch_heap_size parameter. Accuracy decreases as k approaches the size of the heap (which has a default size of 10,000).

The expression approx_topk(k,inner) becomes

topk(
  k,
  eval_cms(
    __count_min_sketch__(inner, shard=1) ++ __count_min_sketch__(inner, shard=2)...
  )
)

__count_min_sketch__ is calculated for each shard and merged on the frontend. Then eval_cms iterates through the labels list and determines the count for each. Then topk selects the top items.

Further resources

Watch: How to turn logs into metrics with Grafana Loki (Loki Community Call July 2025)

LogQL template functions

Thu, 09 Apr 2026 02:28:18 +0000

LogQL template functions

The Go templating language is embedded in the Loki query language, LogQL. The text template format used in | line_format and | label_format support the usage of functions.

Note
In the examples below we use backticks to quote the template strings. This is because some template strings contain double quotes, and using backticks lets us avoid escaping the double quotes. If you are using a different quoting style, you may need to escape the double quotes.

For more information, refer to the Go template documentation.

Additionally you can also access the log line using the __line__ function and the timestamp using the __timestamp__ function.

Template pipeline syntax

A pipeline is a possibly chained sequence of “commands”. A command is a simple value (argument) or a function or method call, possibly with multiple arguments. A pipeline may be “chained” by separating a sequence of commands with pipeline characters ‘|’. In a chained pipeline, the result of each command is passed as the last argument of the following command. The output of the final command in the pipeline is the value of the pipeline.

You can take advantage of pipeline to join together multiple functions.

Example:

`{{ .path | replace " " "_" | trunc 5 | upper }}`

For function that returns a bool such as contains, eq, hasPrefix and hasSuffix, you can apply AND / OR and nested if logic.

Example:

`{{ if and (contains "he" "hello") (contains "llo" "hello") }} yes {{end}}`
`{{ if or (contains "he" "hello") (contains("llo" "hello") }} yes {{end}}`
`{{ if contains "ErrTimeout" .err }} timeout {{else if contains "he" "hello"}} yes {{else}} no {{end}}`

Built-in variables for log line properties

These variables provide a way of referencing something from the log line when writing a template expression.

.label_name

All labels from the Log line are added as variables in the template engine. They can be referenced using the label name prefixed by a .(for example,.label_name). For example the following template will output the value of the path label:

`{{ .path }}`

line

The __line__ function returns the original log line without any modifications.

Signature: line() string

Examples:

`{{ __line__ | lower }}`
`{{ __line__ }}`

timestamp

The __timestamp__ function returns the current log line’s timestamp.

Signature: timestamp() time.Time

`{{ __timestamp__ }}`
`{{ __timestamp__ | date "2006-01-02T15:04:05.00Z-07:00" }}`
`{{ __timestamp__ | unixEpoch }}`

For more information, refer to the blog Parsing and formatting date/time in Go.

Date and time

You can use the following functions to manipulate dates and times when building LogQL queries.

date

Returns a textual representation of the time value formatted according to the provided golang datetime layout.

Signature: date(fmt string, date interface{}) string

Example:

`{{ date "2006-01-02" now }}`

duration

An alias for duration_seconds

Examples:

`{{ .foo | duration }}`
`{{ duration .foo }}`

duration_seconds

Convert a humanized time duration to seconds using time.ParseDuration

Signature: duration_seconds(string) float64

Examples:

`{{ .foo | duration_seconds }}`
`{{ duration_seconds .foo }}`

now

Returns the current time in the local timezone of the Loki server.

Signature: Now() time.Time

Example:

`{{ now }}`

toDate

Parses a formatted string and returns the time value it represents using the local timezone of the server running Loki.

For more consistency between Loki installations, it’s recommended to use toDateInZone

The format string must use the exact date as defined in the golang datetime layout

Signature: toDate(fmt, str string) time.Time

Examples:

`{{ toDate "2006-01-02" "2021-11-02" }}`
`{{ .foo | toDate "2006-01-02T15:04:05.999999999Z" }}`

toDateInZone

Parses a formatted string and returns the time value it represents in the provided timezone.

The format string must use the exact date as defined in the golang datetime layout

The timezone value can be Local, UTC, or any of the IANA Time Zone database values

Signature: toDateInZone(fmt, zone, str string) time.Time

Examples:

`{{ toDateInZone "2006-01-02" "UTC" "2021-11-02" }}`
`{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" }}`

unixEpoch

Returns the number of seconds elapsed since January 1, 1970 UTC.

Signature: unixEpoch(date time.Time) string

Examples:

`{{ unixEpoch now }}`
`{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" | unixEpoch }}`

Example of a query to filter Loki querier jobs which create time is 1 day before:

{job="loki/querier"} | label_format nowEpoch=`{{(unixEpoch now)}}`,createDateEpoch=`{{unixEpoch (toDate "2006-01-02" .createDate)}}` | label_format dateTimeDiff=`{{sub .nowEpoch .createDateEpoch}}` | dateTimeDiff > 86400

unixEpochMillis

Returns the number of milliseconds elapsed since January 1, 1970 UTC.

Signature: unixEpochMillis(date time.Time) string

Examples:

`{{ unixEpochMillis now }}`
`{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" | unixEpochMillis }}`

unixEpochNanos

Returns the number of nanoseconds elapsed since January 1, 1970 UTC.

Signature: unixEpochNanos(date time.Time) string

Examples:

`{{ unixEpochNanos now }}`
`{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" | unixEpochNanos }}`

unixToTime

Converts the string epoch to the time value it represents. Epoch times in days, seconds, milliseconds, microseconds and nanoseconds are supported.

Signature: unixToTime(epoch string) time.Time

Examples:

Consider the following log line {"from": "1679577215","to":"1679587215","message":"some message"}. To print the from field as human readable add the following at the end of the LogQL query:

... | json | line_format `from="{{date "2006-01-02" (unixToTime .from)}}"`

String manipulation

You can use the following templates to manipulate strings when building LogQL Queries.

alignLeft

Use this function to format a string to a fixed with, aligning the content to the left.

Signature: alignLeft(count int, src string) string

Examples:

`{{ alignLeft 5 "hello world"}}` // output: "hello"
`{{ alignLeft 5 "hi"}}`          // output: "hi   "

alignRight

Use this function to format a string to a fixed with, aligning the content to the right.

Signature: alignRight(count int, src string) string

Examples:

`{{ alignRight 5 "hello world"}}` // output: "world"
`{{ alignRight 5 "hi"}}`          // output: "   hi"

b64enc

Base64 encode a string.

Signature: b64enc(string) string

Examples:

`{{ .foo | b64enc }}`
`{{ b64enc  .foo }}`

b64dec

Base64 decode a string.

Signature: b64dec(string) string

Examples:

`{{ .foo | b64dec }}`
`{{ b64dec  .foo }}`

bytes

Convert a humanized byte string to bytes using go-humanize. Durations can be turned into strings such as “3 days ago”, numbers representing sizes like 82854982 into useful strings like, “83 MB” or “79 MiB”

Signature: bytes(string) string

Examples:

`{{ .foo | bytes }}`
`{{ bytes .foo }}`

default

Enables outputting a default value if the source string is otherwise empty. If the ‘src’ parameter is not empty, this function returns the value of ‘src’. Useful for JSON fields that can be missing, like HTTP headers in a log line that aren’t required, as in the following example:

{job="access_log"} | json | line_format `{{.http_request_headers_x_forwarded_for | default "-"}}`

Signature: default(d string, src string) string

Examples:

`{{ default "-" "" }}` // output: -
`{{ default "-" "foo" }}` // output: foo

Example of a query to print a - if the http_request_headers_x_forwarded_for label is empty:

{job="access_log"} | json | line_format `{{.http_request_headers_x_forwarded_for | default "-"}}`

fromJson

Decodes a JSON document into a structure. If the input cannot be decoded as JSON the function will return an empty string.

Signature: fromJson(v string) interface{}

Example:

`{{fromJson "{\"foo\": 55}"}}`

Example of a query to print a newline per queries stored as a json array in the log line:

{job="loki/querier"} |= "finish in prometheus" | logfmt | line_format `{{ range $q := fromJson .queries }} {{ $q.query }} {{ end }}`

lower

Use this function to convert to lower case.

Signature: lower(string) string

Examples:

`{{ .request_method | lower }}`
`{{ lower  "HELLO"}}`

The last example will return hello.

indent

The indent function indents every line in a given string to the specified indent width. This is useful when aligning multi-line strings.

Signature: indent(spaces int,src string) string

Example:

`{{ indent 4 .query }}`

This indents each line contained in the .query by four (4) spaces.

nindent

The nindent function is the same as the indent function, but prepends a new line to the beginning of the string.

Signature: nindent(spaces int,src string) string

Example:

`{{ nindent 4 .query }}`

This will indent every line of text by 4 space characters and add a new line to the beginning.

repeat

Use this function to repeat a string multiple times.

Signature: repeat(c int,value string) string

Example:

`{{ repeat 3 "hello" }}` // output: hellohellohello

printf

Use this function to format a string in a custom way. For more information about the syntax, refer to the Go documentation.

Signature: printf(format string, a ...interface{})

Examples:

`{{ printf "The IP address was %s" .remote_addr }}` // output: The IP address was 129.168.1.1

`{{ printf "%-40.40s" .request_uri}} {{printf "%-5.5s" .request_method}}`
// output: 
// /a/509965767/alternative-to-my-mtg.html  GET
// /id/609259548/hpr.html                   GET

line_format "\"|\" {{printf \"%15.15s\" .ClientHost}} \"|\""

replace

This function performs simple string replacement.

Signature: replace(old string, new string, src string) string

It takes three arguments:

old string to replace
new string to replace with
src source string

Examples:

`{{ .cluster | replace "-cluster" "" }}`
`{{ replace "hello" "world" "hello world" }}`

The last example will return world world.

substr

Get a substring from a string.

Signature: substr(start int,end int,value string) string

If start is < 0, this calls value[:end]. If start is >= 0 and end < 0 or end bigger than s length, this calls value[start:] Otherwise, this calls value[start, end].

Examples:

`{{ .path | substr 2 5 }}`
`{{ substr 0 5 "hello world"}}`  // output: hello
`{{ substr 6 11 "hello world"}}` // output: world

title

Convert to title case.

Signature: title(string) string

Examples:

`{{.request_method | title}}`
`{{ title "hello world"}}`

The last example will return Hello World.

trim

The trim function removes space from either side of a string.

Signature: trim(string) string

Examples:

`{{ .ip | trim }}`
`{{ trim "   hello    " }}` // output: hello

trimAll

Use this function to remove given characters from the front or back of a string.

Signature: trimAll(chars string,src string) string

Examples:

`{{ .path | trimAll "/" }}`
`{{ trimAll "$" "$5.00" }}` // output: 5.00

trimPrefix

Use this function to trim just the prefix from a string.

Signature: trimPrefix(prefix string, src string) string

Examples:

`{{  .path | trimPrefix "/" }}`
`{{ trimPrefix "-" "-hello" }}` // output: hello

trimSuffix

Use this function to trim just the suffix from a string.

Signature: trimSuffix(suffix string, src string) string

Examples:

`{{  .path | trimSuffix "/" }}`
`{{ trimSuffix "-" "hello-" }}` // output: hello

trunc

Truncate a string and add no suffix.

Signature: trunc(count int,value string) string

Examples:

`{{ .path | trunc 2 }}`
`{{ trunc 5 "hello world"}}`   // output: hello
`{{ trunc -5 "hello world"}}`  // output: world

upper

Use this function to convert to upper case.

Signature: upper(string) string

Examples:

`{ .request_method | upper }}`
`{{ upper  "hello"}}`

This results in HELLO.

urlencode

Use this function to urlencode a string.

Signature: urlencode(string) string

Examples:

`{{ .request_url | urlencode }}`
`{{ urlencode  .request_url}}`

urldecode

Use this function to urldecode a string.

Signature: urldecode(string) string

Examples:

`{{ .request_url | urldecode }}`
`{{ urldecode  .request_url}}`

Logical functions

You can use the following logical functions to compare strings when building a template expression.

contains

Use this function to test to see if one string is contained inside of another.

Signature: contains(s string, src string,) bool

Examples:

`{{ if contains "ErrTimeout" .err }} timeout {{end}}`
`{{ if contains "he" "hello" }} yes {{end}}`

eq

Use this function to test to see if one string has exact matching inside of another.

Signature: eq(s string, src string) bool

Examples:

`{{ if eq "ErrTimeout" .err }} timeout {{end}}`
`{{ if eq "hello" "hello" }} yes {{end}}`

hasPrefix and hasSuffix

The hasPrefix and hasSuffix functions test whether a string has a given prefix or suffix.

Signatures:

hasPrefix(prefix string, src string) bool
hasSuffix(suffix string, src string) bool

Examples:

`{{ if hasSuffix .err "Timeout" }} timeout {{end}}`
`{{ if hasPrefix "he" "hello" }} yes {{end}}`

Mathematical functions

You can use the following mathematical functions when writing template expressions.

add

Sum numbers. Supports multiple numbers

Signature: func(i ...interface{}) int64

Example:

`{{ add 3 2 5 }}` // output: 10

addf

Sum floating point numbers. Supports multiple numbers.

Signature: func(i ...interface{}) float64

Example:

`{{ addf 3.5 2 5 }}` // output: 10.5

ceil

Returns the greatest float value greater than or equal to input value

Signature: ceil(a interface{}) float64

Example:

`{{ ceil 123.001 }}` //output 124.0

div

Divide two integers.

Signature: func(a, b interface{}) int64

Example:

`{{ div 10 2}}` // output: 5

divf

Divide floating point numbers. Supports multiple numbers.

Signature: func(a interface{}, v ...interface{}) float64

Example:

`{{ divf 10 2 4}}` // output: 1.25

float64

Convert a string to a float64.

Signature: toFloat64(v interface{}) float64

Example:

`{{ "3.5" | float64 }}` //output 3.5

floor

Returns the greatest float value less than or equal to input value.

Signature: floor(a interface{}) float64

Example:

`{{ floor 123.9999 }}` //output 123.0

int

Convert value to an integer.

Signature: toInt(v interface{}) int

Example:

`{{ "3" | int }}` //output 3

max

Return the largest of a series of integers:

Signature: max(a interface{}, i ...interface{}) int64

Example:

`{{ max 1 2 3 }}` //output 3

maxf

Return the largest of a series of floats:

Signature: maxf(a interface{}, i ...interface{}) float64

Example:

`{{ maxf 1 2.5 3 }}` //output 3

min

Return the smallest of a series of integers.

Signature: min(a interface{}, i ...interface{}) int64

Example:

`{{ min 1 2 3 }}`//output 1

minf

Return the smallest of a series of floats.

Signature: minf(a interface{}, i ...interface{}) float64

Example:

`{{ minf 1 2.5 3 }}` //output 1.5

mul

Multiply numbers. Supports multiple numbers.

Signature: mul(a interface{}, v ...interface{}) int64

Example:

`{{ mul 5 2 3}}` // output: 30

mulf

Multiply floating numbers. Supports multiple numbers

Signature: mulf(a interface{}, v ...interface{}) float64

Example:

`{{ mulf 5.5 2 2.5 }}` // output: 27.5

mod

Returns the remainder when number ‘a’ is divided by number ‘b’.

Signature: mod(a, b interface{}) int64

Example:

`{{ mod 10 3}}` // output: 1

round

Returns a float value with the remainder rounded to the given number of digits after the decimal point.

Signature: round(a interface{}, p int, rOpt ...float64) float64

Example:

`{{ round 123.555555 3 }}` //output 123.556

We can also provide a roundOn number as third parameter

Example:

`{{ round 123.88571428571 5 .2 }}` //output 123.88572

With default roundOn of .5 the above value would be 123.88571

sub

Subtract one number from another.

Signature: func(a, b interface{}) int64

Example:

`{{ sub 5 2 }}` // output: 3

subf

Subtract floating numbers. Supports multiple numbers.

Signature: func(a interface{}, v ...interface{}) float64

Example:

`{{ subf  5.5 2 1.5 }}` // output: 2

Regular expression functions

You can use the following functions to perform regular expressions in a template expression.

count

Counts occurrences of the regex (regex) in (src).

Signature: count(regex string, src string) int

Examples:

`{{ count "a|b" "abab" }}` // output: 4
`{{ count "o" "foo" }}`    // output: 2

Example of a query to print how many times XYZ occurs in a line:

{job="xyzlog"} | line_format `{{ __line__ | count "XYZ"}}`

regexReplaceAll and regexReplaceAllLiteral

regexReplaceAll returns a copy of the input string, replacing matches of the Regexp with the replacement string replacement. Inside string replacement, $ signs are interpreted as in Expand, so for instance $1 represents the text of the first sub-match. See the golang Regexp.replaceAll documentation for more examples.

Signature: regexReplaceAll(regex string, src string, replacement string) (source)

Example:

`{{ regexReplaceAll "(a*)bc" .some_label "${1}a" }}`

regexReplaceAllLiteral function returns a copy of the input string and replaces matches of the Regexp with the replacement string replacement. The replacement string is substituted directly, without using Expand.

Signature: regexReplaceAllLiteral(regex string, src string, replacement string)

Example:

`{{ regexReplaceAllLiteral "(ts=)" .timestamp "timestamp=" }}`

LogCLI

Thu, 09 Apr 2026 02:28:18 +0000

LogCLI

LogCLI is a command-line tool for querying and exploring logs in Grafana Loki:

Installation and Reference: LogCLI
Getting started tutorial: LogCLI Tutorial

Matching IP addresses

Thu, 09 Apr 2026 02:28:18 +0000

Matching IP addresses

LogQL supports matching IP addresses.

With logs such as

3.180.71.3 - - [17/May/2015:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"
80.91.33.133 - - [17/May/2015:08:05:14 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.16)"
46.4.66.76 - - [17/May/2015:08:05:45 +0000] "GET /downloads/product_1 HTTP/1.1" 404 318 "-" "Debian APT-HTTP/1.3 (1.0.1ubuntu2)"
93.180.71.3 - - [17/May/2015:08:05:26 +0000] "GET /downloads/product_1 HTTP/1.1" 404 324 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"

the LogQL line filter is not sufficient. A line filter such as

{job_name="myapp"} |= "3.180.71.3"

also matches example IP addresses such as 93.180.71.3. A better choice uses a regexp: |~"^3.180.71.3". This regexp does not handle IPv6 addresses, and it does not match a range of IP addresses.

The LogQL support for matching IP addresses handles both IPv4 and IPv6 single addresses, as well as ranges within IP addresses and CIDR patterns.

Match IP addresses with the syntax: ip("<pattern>"). The <pattern> can be:

A single IP address. Examples: ip("192.0.2.0"), ip("::1")
A range within the IP address. Examples: ip("192.168.0.1-192.189.10.12"), ip("2001:db8::1-2001:db8::8")
A CIDR specification. Examples: ip("192.51.100.0/24"), ip("2001:db8::/32")

The IP matching can be used in both line filter and label filter expressions. When specifying line filter expressions, only the |= and != operations are allowed. When specifying label filter expressions, only the = and != operations are allowed.

Line filter examples

{job_name="myapp"} |= ip("192.168.4.5/16")

Return log lines that do not match with an IPv4 range:

{job_name="myapp"} != ip("192.168.4.5-192.168.4.20")

Label filter examples
logql
```
{job_name="myapp"}
	| logfmt
	| remote_addr = ip("2001:db8::1-2001:db8::8")
	| level = "error"
```
Filters can also be chained. This example matches log lines with all IPv4 subnet values 192.168.4.5/16 except IP address 192.168.4.2:
logql
```
{job_name="myapp"}
	| logfmt
	| addr = ip("192.168.4.5/16")
	| addr != ip("192.168.4.2")
```
This example use the conditional or and matches log lines with either, all IPv4 subnet values 192.168.4.0/24 OR all IPv4 subnet values 10.10.15.0/24:
logql
```
{job_name="myapp"}
	| logfmt
	| addr = ip("192.168.4.0/24") or addr = ip("10.10.15.0/24")
```

Query examples

Thu, 09 Apr 2026 02:28:18 +0000

Query examples

These LogQL query examples have explanations of what the queries accomplish.

Log query examples

Examples that filter on IP address

Return log lines that are not within a range of IPv4 addresses:
logql
```
{job_name="myapp"} != ip("192.168.4.5-192.168.4.20")
```
This example matches log lines with all IPv4 subnet values 192.168.4.5/16 except IP address 192.168.4.2:
logql
```
{job_name="myapp"}
	| logfmt
	| addr = ip("192.168.4.5/16")
	| addr != ip("192.168.4.2")
```

Examples that aid in security evaluation

Extract the user and IP address of failed logins from Linux /var/log/secure

{job="security"} 
    |~ "Invalid user.*"
    | regexp "(^(?P<user>\\S+ {1,2}){8})"
    | regexp "(^(?P<ip>\\S+ {1,2}){10})"
    | line_format "IP = {{.ip}}\tUSER = {{.user}}"

Get successful logins from Linux /var/log/secure

{job="security"}
    != "grafana_com"
    |= "session opened"
    != "sudo: "
    | regexp "(^(?P<user>\\S+ {1,2}){11})"
    | line_format "USER = {{.user}}"

Metrics query examples

Return the per-second rate of all non-timeout errors within the last minutes per host for the MySQL job, and only include errors whose duration is above ten seconds.
```
sum by (host) (rate({job="mysql"}
    |= "error" != "timeout"
    | json
    | duration > 10s [1m]))
```

Multiple filtering stages examples

Query results are gathered by successive evaluation of parts of the query from left to right. To make querying efficient, order the filtering stages left to right:

stream selector
line filters
label filters

Consider the query:

{cluster="ops-tools1", namespace="loki-dev", job="loki-dev/query-frontend"} |= "metrics.go" != "out of order" | logfmt | duration > 30s or status_code != "200"

Within this query, the stream selector is

{cluster="ops-tools1", namespace="loki-dev", job="loki-dev/query-frontend"}

There are two line filters: |= "metrics.go" and !="out of order". Of the log lines identified with the stream selector, the query results include only those log lines that contain the string “metrics.go” and do not contain the string “out of order”.

The logfmt parser produces the duration and status_code labels, such that they can be used by a label filter.

The label filter | duration > 30s or status_code!="200" further filters out log lines. It includes those log lines that contain a status_code label with any value other than the value 200, as well as log lines that contain a duration label with a value greater than 30 sections,

While every query will have a stream selector, not all queries will have line and label filters.

Examples that use multiple parsers

Consider this logfmt log line. To extract the method and the path of this logfmt log line,

level=debug ts=2020-10-02T10:10:42.092268913Z caller=logging.go:66 traceID=a9d4d8a928d8db1 msg="POST /api/prom/api/v1/query_range (200) 1.5s"

To extract the method and the path, use multiple parsers (logfmt and regexp):

{job="loki-ops/query-frontend"} | logfmt | line_format "{{.msg}}" | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \\((?P<status>\\d+?)\\) (?P<duration>.*)"

This is possible because the | line_format reformats the log line to become POST /api/prom/api/v1/query_range (200) 1.5s which can then be parsed with the | regexp ... parser.

Log line formatting examples

The following query shows how you can reformat a log line to make it easier to read on screen.

{cluster="ops-tools1", name="querier", namespace="loki-dev"}
  |= "metrics.go" != "loki-canary"
  | logfmt
  | query != ""
  | label_format query="{{ Replace .query \"\\n\" \"\" -1 }}"
  | line_format "{{ .ts}}\t{{.duration}}\ttraceID = {{.traceID}}\t{{ printf \"%-100.100s\" .query }} "

Label formatting is used to sanitize the query while the line format reduce the amount of information and creates a tabular output.

For these given log lines:

level=info ts=2020-10-23T20:32:18.094668233Z caller=metrics.go:81 org_id=29 traceID=1980d41501b57b68 latency=fast query="{cluster=\"ops-tools1\", job=\"loki-ops/query-frontend\"} |= \"query_range\"" query_type=filter range_type=range length=15m0s step=7s duration=650.22401ms status=200 throughput_mb=1.529717 total_bytes_mb=0.994659
level=info ts=2020-10-23T20:32:18.068866235Z caller=metrics.go:81 org_id=29 traceID=1980d41501b57b68 latency=fast query="{cluster=\"ops-tools1\", job=\"loki-ops/query-frontend\"} |= \"query_range\"" query_type=filter range_type=range length=15m0s step=7s duration=624.008132ms status=200 throughput_mb=0.693449 total_bytes_mb=0.432718

The result would be:

2020-10-23T20:32:18.094668233Z	650.22401ms	    traceID = 1980d41501b57b68	{cluster="ops-tools1", job="loki-ops/query-frontend"} |= "query_range"
2020-10-23T20:32:18.068866235Z	624.008132ms	traceID = 1980d41501b57b68	{cluster="ops-tools1", job="loki-ops/query-frontend"} |= "query_range"

It’s possible to strip ANSI sequences from the log line, making it easier to parse it further:

{job="example"} | decolorize

This way this log line:

[2022-11-04 22:17:57.811] \033[0;32http\033[0m: GET /_health (0 ms) 204

turns into:

[2022-11-04 22:17:57.811] http: GET /_health (0 ms) 204

Unwrap examples

Calculate the p99 of the nginx-ingress latency by path:

quantile_over_time(0.99,
  {cluster="ops-tools1",container="ingress-nginx"}
    | json
    | __error__ = ""
    | unwrap request_time [1m]) by (path)

Calculate the quantity of bytes processed per organization ID:

sum by (org_id) (
  sum_over_time(
  {cluster="ops-tools1",container="loki-dev"}
      |= "metrics.go"
      | logfmt
      | unwrap bytes_processed [1m])
  )

Vector aggregation examples

Get the top 10 applications by the highest log throughput:

topk(10,sum(rate({region="us-east1"}[5m])) by (name))

Get the count of log lines for the last five minutes for a specified job, grouping by level:

sum(count_over_time({job="mysql"}[5m])) by (level)

Get the rate of HTTP GET requests to the /home endpoint for NGINX logs by region:

avg(rate(({job="nginx"} |= "GET" | json | path="/home")[10s])) by (region)

Query acceleration

Thu, 09 Apr 2026 02:28:18 +0000

Query acceleration

Warning
In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is an experimental feature. Engineering and on-call support is not available. No SLA is provided.

In Grafana Cloud, Query acceleration using Bloom filters is enabled as a public preview for select large-scale customers that are ingesting more that 75TB of logs a month. Limited support and no SLA are provided.

If bloom filters are enabled, you can write LogQL queries using structured metadata to benefit from query acceleration.

Prerequisites

Bloom filters must be enabled. OpenSource and Enterprise customers must enable bloom filters. Grafana Support enables Bloom Filters for Cloud Customers who are part of the public preview.
Logs must be sending structured metadata. OpenSource and Enterprise customers must enable structured metadata. Structured metadata is enabled by default in Grafana Cloud.

Query blooms

Queries will be accelerated for any label filter expression that satisfies all of the following criteria:

The label filter expression using string equality, such as | key="value".
- or and and operators can be used to match multiple values, such as | detected_level="error" or detected_level="warn".
- Basic regular expressions are automatically simplified into a supported expression:
  - | key=~"value" is converted to | key="value".
  - | key=~"value1|value2" is converted to | key="value1" or key="value2".
  - | key=~".+" checks for existence of key. .* is not supported.
The label filter expression is querying for structured metadata and not a stream label.
The label filter expression is placed before any parser expression, labels format expression, drop labels expression, or keep labels expression.

To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parser expression:

In the following example, the query is not accelerated because the structured metadata filter, detected_level="error", is after a parser stage, json.

{cluster="prod"} | logfmt | json | detected_level="error"

In the following example, the query is accelerated because the structured metadata filter is before any parser stage.

{cluster="prod"} | detected_level="error" | logfmt | json

LogQL Reference

Thu, 09 Apr 2026 02:28:18 +0000

LogQL Reference

Loki accepts a number of operators and functions within a LogQL query.

Binary operators

Arithmetic operators

The following binary arithmetic operators exist in Loki:

+ (addition)
- (subtraction)
* (multiplication)
/ (division)
% (modulo)
^ (power/exponentiation)

Binary arithmetic operators are defined between two literals (scalars), a literal and a vector, and two vectors.

Between two literals, the behavior is obvious: They evaluate to another literal that is the result of the operator applied to both scalar operands (1 + 1 = 2).

Between a vector and a literal, the operator is applied to the value of every data sample in the vector, e.g. if a time series vector is multiplied by 2, the result is another vector in which every sample value of the original vector is multiplied by 2.

Between two vectors, a binary arithmetic operator is applied to each entry in the left-hand side vector and its matching element in the right-hand vector. The result is propagated into the result vector with the grouping labels becoming the output label set. Entries for which no matching entry in the right-hand vector can be found are not part of the result.

Pay special attention to operator order when chaining arithmetic operators.

Arithmetic Examples

Implement a health check with a simple query:

1 + 1

Double the rate of a log stream’s entries:

sum(rate({app="foo"}[1m])) * 2

Get proportion of warning logs to error logs for the foo app

sum(rate({app="foo", level="warn"}[1m])) / sum(rate({app="foo", level="error"}[1m]))

Logical and set operators

These logical/set binary operators are only defined between two vectors:

and (intersection)
or (union)
unless (complement)

vector1 and vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped.

vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.

vector1 unless vector2 results in a vector consisting of the elements of vector1 for which there are no elements in vector2 with exactly matching label sets. All matching elements in both vectors are dropped.

Binary operators examples

This contrived query will return the intersection of these queries, effectively rate({app="bar"}):

rate({app=~"foo|bar"}[1m]) and rate({app="bar"}[1m])

Comparison operators

== (equality)
!= (inequality)
> (greater than)
>= (greater than or equal to)
< (less than)
<= (less than or equal to)

Comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. By default they filter. Their behavior can be modified by providing bool after the operator, which will return 0 or 1 for the value rather than filtering.

Between two scalars, these operators result in another scalar that is either 0 (false) or 1 (true), depending on the comparison result. The bool modifier must not be provided.

1 >= 1 is equivalent to 1

Between a vector and a scalar, these operators are applied to the value of every data sample in the vector, and vector elements between which the comparison result is false get dropped from the result vector. If the bool modifier is provided, vector elements that would be dropped instead have the value 0 and vector elements that would be kept have the value 1.

Filters the streams which logged at least 10 lines in the last minute:

count_over_time({foo="bar"}[1m]) > 10

Attach the value(s) 0/1 to streams that logged less/more than 10 lines:

count_over_time({foo="bar"}[1m]) > bool 10

Between two vectors, these operators behave as a filter by default, applied to matching entries. Vector elements for which the expression is not true or which do not find a match on the other side of the expression get dropped from the result, while the others are propagated into a result vector. If the bool modifier is provided, vector elements that would have been dropped instead have the value 0 and vector elements that would be kept have the value 1, with the grouping labels again becoming the output label set.

Return the streams matching app=foo without app labels that have higher counts within the last minute than their counterparts matching app=bar without app labels:

sum without(app) (count_over_time({app="foo"}[1m])) > sum without(app) (count_over_time({app="bar"}[1m]))

Same as above, but vectors have their values set to 1 if they pass the comparison or 0 if they fail/would otherwise have been filtered out:

sum without(app) (count_over_time({app="foo"}[1m])) > bool sum without(app) (count_over_time({app="bar"}[1m]))

Pattern match filter operators

|> (line match pattern)
!> (line match not pattern)

Pattern Filter not only enhances efficiency but also simplifies the process of writing LogQL queries. By eliminating the need for complex regex patterns, users can create queries using a more intuitive syntax, reducing the cognitive load and potential for errors.

Within the pattern syntax the <_> serves as a wildcard, representing any arbitrary text. This allows the query to match log lines where the specified pattern occurs, such as log lines containing static content, with variable content in between.

Line match pattern example:

{service_name=`distributor`} |> `<_> caller=http.go:194 level=debug <_> msg="POST /push.v1.PusherService/Push <_>`

Line match not pattern example:

{service_name=`distributor`} !> `<_> caller=http.go:194 level=debug <_> msg="POST /push.v1.PusherService/Push <_>`

For example, the example queries above will respectively match and not match the following log line from the distributor service:

ts=2024-04-05T08:40:13.585911094Z caller=http.go:194 level=debug traceID=23e54a271db607cc orgID=3648 msg="POST /push.v1.PusherService/Push (200) 12.684035ms"
ts=2024-04-05T08:41:06.551403339Z caller=http.go:194 level=debug traceID=54325a1a15b42e2d orgID=1218 msg="POST /push.v1.PusherService/Push (200) 1.664285ms"
ts=2024-04-05T08:41:06.506524777Z caller=http.go:194 level=debug traceID=69d4271da1595bcb orgID=1218 msg="POST /push.v1.PusherService/Push (200) 1.783818ms"
ts=2024-04-05T08:41:06.473740396Z caller=http.go:194 level=debug traceID=3b8ec973e6397814 orgID=3648 msg="POST /push.v1.PusherService/Push (200) 1.893987ms"
ts=2024-04-05T08:41:05.88999067Z caller=http.go:194 level=debug traceID=6892d7ef67b4d65c orgID=3648 msg="POST /push.v1.PusherService/Push (200) 2.314337ms"
ts=2024-04-05T08:41:05.826266414Z caller=http.go:194 level=debug traceID=0bb76e910cfd008d orgID=3648 msg="POST /push.v1.PusherService/Push (200) 3.625744ms"

Order of operations

When chaining or combining operators, you have to consider operator precedence: Generally, you can assume regular mathematical convention with operators on the same precedence level being left-associative.

More details can be found in the Golang language documentation.

1 + 2 / 3 is equal to 1 + ( 2 / 3 ).

2 * 3 % 2 is evaluated as (2 * 3) % 2.

Keywords on and ignoring

The ignoring keyword causes specified labels to be ignored during matching. The syntax:

<vector expr> <bin-op> ignoring(<labels>) <vector expr>

This example will return the machines which total count within the last minutes exceed average value for app foo.

max by(machine) (count_over_time({app="foo"}[1m])) > bool ignoring(machine) avg(count_over_time({app="foo"}[1m]))

The on keyword reduces the set of considered labels to a specified list. The syntax:

<vector expr> <bin-op> on(<labels>) <vector expr>

This example will return every machine total count within the last minutes ratio in app foo:

sum by(machine) (count_over_time({app="foo"}[1m])) / on() sum(count_over_time({app="foo"}[1m]))

Many-to-one and one-to-many vector matches

Many-to-one and one-to-many matchings occur when each vector element on the “one”-side can match with multiple elements on the “many”-side. You must explicitly request matching by using the group_left or group_right modifier, where left or right determines which vector has the higher cardinality. The syntax:

<vector expr> <bin-op> ignoring(<labels>) group_left(<labels>) <vector expr>
<vector expr> <bin-op> ignoring(<labels>) group_right(<labels>) <vector expr>
<vector expr> <bin-op> on(<labels>) group_left(<labels>) <vector expr>
<vector expr> <bin-op> on(<labels>) group_right(<labels>) <vector expr>

The label list provided with the group modifier contains additional labels from the “one”-side that are included in the result metrics. And a label should only appear in one of the lists specified by on and group_x. Every time series of the result vector must be uniquely identifiable. Grouping modifiers can only be used for comparison and arithmetic. By default, the system matches and, unless, and or operations with all entries in the right vector.

The following example returns the rates requests partitioned by app and status as a percentage of total requests.

sum by (app, status) (
  rate(
    {job="http-server"}
      | json
      [5m]
  )
)
/ on (app) group_left
sum by (app) (
  rate(
    {job="http-server"}
      | json
      [5m]
  )
)

=>
[
  {app="foo", status="200"} => 0.8
  {app="foo", status="400"} => 0.1
  {app="foo", status="500"} => 0.1
]

This version uses group_left(<labels>) to include <labels> from the right hand side in the result and returns the cost of discarded events per user, organization, and namespace:

sum by (user, namespace) (
  rate(
    {job="events"}
      | logfmt
      | discarded="true"
      [5m]
  )
)
* on (user) group_left(organization)
max_over_time(
  {job="cost-calculator"}
    | logfmt
    | unwrap cost
    [5m]
) by (user, organization)

=>
[
  {user="foo", namespace="dev", organization="little-org"} => 10
  {user="foo", namespace="prod", organization="little-org"} => 50
  {user="bar", namespace="dev", organization="big-org"} => 70
  {user="bar", namespace="prod", organization="big-org"} => 200
]

Comments

LogQL queries can be commented using the # character:

{app="foo"} # anything that comes after will not be interpreted in your query

With multi-line LogQL queries, the query parser can exclude whole or partial lines using #:

{app="foo"}
    | json
    # this line will be ignored
    | bar="baz" # this checks if bar = "baz"

Pipeline Errors

There are multiple reasons which cause pipeline processing errors, such as:

A numeric label filter may fail to turn a label value into a number
A metric conversion for a label may fail.
A log line is not a valid json document.
etc…

When those failures happen, Loki won’t filter out those log lines. Instead they are passed into the next stage of the pipeline with a new system label named __error__. The only way to filter out errors is by using a label filter expressions. The __error__ label can’t be renamed via the language.

For example to remove json errors:

  {cluster="ops-tools1",container="ingress-nginx"}
    | json
    | __error__ != "JSONParserErr"

Alternatively you can remove all error using a catch all matcher such as __error__ = "" or even show only errors using __error__ != "".

The filter should be placed after the stage that generated this error. This means if you need to remove errors from an unwrap expression it needs to be placed after the unwrap.

quantile_over_time(
	0.99,
	{container="ingress-nginx",service="hosted-grafana"}
	| json
	| unwrap response_latency_seconds
	| __error__=""[1m]
	) by (cluster)

Metric queries cannot contain errors, in case errors are found during execution, Loki will return an error and appropriate status code.

Functions

Loki supports functions to operate on data.

label_replace()

For each time series in v,

label_replace(v instant-vector,
    dst_label string,
    replacement string,
    src_label string,
    regex string)

matches the regular expression regex against the label src_label. If it matches, then the time series is returned with the label dst_label replaced by the expansion of replacement.

$1 is replaced with the first matching subgroup, $2 with the second etc. If the regular expression doesn’t match, then the time series is returned unchanged.

This example will return a vector with each time series having a foo label with the value a added to it:

label_replace(rate({job="api-server",service="a:c"} |= "err" [1m]), "foo", "$1",
  "service", "(.*):.*")

Troubleshoot log queries (READ)

Thu, 09 Apr 2026 02:28:18 +0000

Troubleshoot log queries (READ)

This guide helps you troubleshoot errors that occur when querying logs from Loki. When Loki rejects or fails query requests, it’s typically due to query syntax errors, exceeding limits, timeout issues, or storage access problems.

Before you begin, ensure you have the following:

Access to Grafana Loki logs and metrics
Understanding of LogQL query language basics
Permissions to configure limits and settings if needed

Monitoring query errors

Query errors can be observed using these Prometheus metrics:

loki_request_duration_seconds - Query latency by route and status code
loki_logql_querystats_bytes_processed_per_seconds - Bytes processed during queries
loki_frontend_query_range_duration_seconds_bucket - Frontend query latency

You can set up alerts on 4xx and 5xx status codes to detect query problems early. This can be helpful when tuning limits configurations.

LogQL parse errors

Parse errors occur when the LogQL query syntax is invalid. Loki returns HTTP status code 400 Bad Request for all parse errors.

Error: Failed to parse the log query

Error message:

failed to parse the log query

Or with position details:

parse error at line <line>, col <col>: <message>

Cause:

The LogQL query contains syntax errors. This could be due to:

Missing or mismatched brackets, quotes, or braces
Invalid characters or operators
Incorrect function syntax
Invalid duration format

Common examples:

Invalid Query	Error	Fix
`{app="foo"`	Missing closing brace	`{app="foo"}`
`{app="foo"} \|= test`	Unquoted filter string	`{app="foo"} \|= "test"`
`rate({app="foo"}[5minutes])`	Invalid duration unit	`rate({app="foo"}[5m])`
`{app="foo"} \| json`	Missing pipe symbol before parser	`{app="foo"} \| json`

Resolution:

Start with a simple stream selector: {job="app"}, then add filters and operations incrementally to identify syntax issues.

Check bracket matching - Ensure all {, }, (, ), [, ] are properly closed.
Verify string quoting - All label values and filter strings must be quoted.
Use valid duration units - Use ns, us, ms, s, m, h, d, w, y , for example, 5m not 5minutes.
Review operator syntax - Ensure label matchers use proper operators (=, !=, =~, !~). Check the LogQL documentation for correct operator usage.
Use Grafana Assistant - If you are a Cloud Logs user, you can use Grafana Assistant to write or revise your query using natural language, for example, “What errors occurred for application foo in the last hour?”

Properties:

Enforced by: Query Frontend/Querier
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: At least one equality matcher required

Error message:

parse error : queries require at least one regexp or equality matcher that does not have an empty-compatible value. For instance, app=~".*" does not meet this requirement, but app=~".+" will

Cause:

The query uses only negative matchers (!=, !~) or matchers that match empty strings (=~".*"), which would select all streams. This is prevented to protect against accidentally querying the entire database.

Invalid examples:

{foo!="bar"}
{app=~".*"}
{foo!~"bar|baz"}

Valid examples:

{foo="bar"}
{app=~".+"}
{app="baz", foo!="bar"}

Resolution:

Add at least one positive matcher that selects specific streams.
Use .+ instead of .* in regex matchers to require at least one character.
Add additional label selectors to narrow down the query scope.
Use Grafana Assistant - If you are a Cloud Logs user, you can use Grafana Assistant to write or revise your query using natural language, for example, “Find logs containing ’foo’ but not ‘bar’ or ‘baz’.”

Properties:

Enforced by: Query Frontend
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Only label matchers are supported

Error message:

only label matchers are supported

Cause:

The query was passed to an API that only accepts label matchers (like the series API), but included additional expressions like line filters or parsers.

Resolution:

Use only stream selectors for APIs that don’t support full LogQL:

# Valid for series API
{app="foo", env="prod"}

# Invalid for series API
{app="foo"} |= "error"

Properties:

Enforced by: API handler
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Log queries not supported as instant query type

Error message:

log queries are not supported as an instant query type, please change your query to a range query type

Cause:

A log query (one that returns log lines rather than metrics) was submitted to the instant query endpoint (/loki/api/v1/query). Log queries must use the range query endpoint.

Resolution:

Convert to a range query Convert log queries to range queries with a time range. Range queries are the default in Grafana Explore.
Use the range query endpoint /loki/api/v1/query_range for log queries.

Convert to a metric query if you need to use instant queries:

# This is a log query (returns logs)
{app="foo"} |= "error"

# This is a metric query (can be instant)
count_over_time({app="foo"} |= "error"[5m])

Use Grafana Assistant - If you are a Cloud Logs user, you can use Grafana Assistant to write or revise your query.

Properties:

Enforced by: Query API
Retryable: No (use correct endpoint or query type)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Invalid aggregation without unwrap

Error message:

parse error : invalid aggregation sum_over_time without unwrap

Cause:

Aggregation functions like sum_over_time, avg_over_time, min_over_time, max_over_time require an unwrap expression to extract a numeric value from log lines.

Resolution:

Add an unwrap expression to extract the numeric label:

# Invalid
sum_over_time({app="foo"} | json [5m])

# Valid - unwrap a numeric label
sum_over_time({app="foo"} | json | unwrap duration [5m])

Properties:

Enforced by: Query Parser
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Invalid aggregation with unwrap

Error message:

parse error : invalid aggregation count_over_time with unwrap

Cause:

The count_over_time function doesn’t use unwrapped values - it just counts log lines. Using it with unwrap is invalid.

Resolution:

Remove the unwrap expression for count_over_time:

# Invalid
count_over_time({app="foo"} | json | unwrap duration [5m])

# Valid
count_over_time({app="foo"} | json [5m])

Use sum_over_time if you want to sum unwrapped values.

Properties:

Enforced by: Query Parser
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Query limit errors

These errors occur when queries exceed configured resource limits. They return HTTP status code 400 Bad Request.

Error: Maximum series reached

Error message:

maximum number of series (<limit>) reached for a single query; consider reducing query cardinality by adding more specific stream selectors, reducing the time range, or aggregating results with functions like sum(), count() or topk()

Cause:

The query matches more unique label combinations (series) than the configured limit allows. This protects against queries that would consume excessive memory.

Default configuration:

max_query_series: 500 (default)

Resolution:

Add more specific stream selectors to reduce cardinality:

# Too broad
{job="ingress-nginx"}

# More specific
{job="ingress-nginx", namespace="production", pod=~"ingress-nginx-.*"}

Reduce the time range of the query.
Use label filters to narrow down results: {job="app"} |= "error"
Use aggregation functions to reduce cardinality:
logql
```
sum by (status) (rate({job="nginx"} | json [5m]))
```

Increase the limit if resources allow:

limits_config:
  max_query_series: 1000  #default is 500

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Cardinality issues

Error message:

cardinality limit exceeded for {}; 100001 entries, more than limit of 100000

Cause:

The query produces results with too many unique label combinations. This protects against queries that would generate excessive memory usage and slow performance.

Default configuration:

cardinality_limit: 100000

Resolution:

Use more specific label selectors to reduce the number of unique streams.
Apply aggregation functions to reduce cardinality:
logql
```
sum by (status) (rate({job="nginx"}[5m]))
```

Use by() or without() clauses to group results and reduce dimensions:

sum by (status, method) (rate({job="nginx"} | json [5m]))

Another alternative is using drop or keep to reduce the number of labels and hence the cardinality:

# Drop high-cardinality labels like request_id or trace_id
{job="nginx"} | json | drop request_id, trace_id, session_id

# Keep only the labels you need
{job="nginx"} | json | keep status, method, path

Increase the limit if needed:

limits_config:
  cardinality_limit: 200000  #default is 100000

Properties:

Enforced by: Query Engine
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Max entries limit per query exceeded

Error message:

max entries limit per query exceeded, limit > max_entries_limit_per_query (<requested> > <limit>)

Cause:

The query requests more log entries than the configured maximum. This applies to log queries (not metric queries).

Default configuration:

max_entries_limit_per_query: 5000

Resolution:

Reduce the limit parameter in your query request.
Add more specific filters to return fewer results:
logql
```
{app="foo"} |= "error" 
```
Reduce the time range of the query.

Increase the limit if needed:

limits_config:
  max_entries_limit_per_query: 10000  #default is 5000

Properties:

Enforced by: Querier/Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Query would read too many bytes

Error message:

the query would read too many bytes (query: <size>, limit: <limit>); consider adding more specific stream selectors or reduce the time range of the query

Cause:

The estimated data volume for the query exceeds the configured limit. This is determined before query execution using index statistics.

Default configuration:

max_query_bytes_read: 0B (disabled by default)

Resolution:

Add more specific stream selectors to reduce data volume.
Reduce the time range of the query.
Increase the limit if resources allow:
YAML
```
limits_config:
  max_query_bytes_read: 10GB
```

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Too many chunks (count)

Error message:

the query hit the max number of chunks limit (limit: 2000000 chunks)

Cause:

The number of chunks that the query would read exceeds the configured limit. This protects against queries that would scan excessive amounts of data and consume too much memory.

Default configuration:

max_chunks_per_query: 2000000

Resolution:

Narrow stream selectors to reduce the number of matching chunks:

# Too broad
{job="app"}

# More specific
{job="app", environment="production", namespace="api"}

Reduce the query time range to scan fewer chunks.

Increase the limit if resources allow:

limits_config:
  max_chunks_per_query: 5000000  #default is 2000000

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Stream matcher limits

Error message:

max streams matchers per query exceeded, matchers-count > limit (1500 > 1000)

Cause:

The query contains too many stream matchers. This limit prevents queries with excessive complexity that could impact query performance.

Default configuration:

max_streams_matchers_per_query: 1000

Resolution:

Simplify your query by using fewer label matchers.
Combine multiple queries instead of using many OR conditions.

Use regex matchers to consolidate multiple values:

# Good: 3 matchers using regex patterns
{cluster="prod", namespace=~"api|web", pod=~"nginx-.*"}

Increase the limit if needed:

limits_config:
  max_streams_matchers_per_query: 2000  #default is 1000

Properties:

Enforced by: Querier
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Query too large for single querier

Error message:

query too large to execute on a single querier: (query: <size>, limit: <limit>); consider adding more specific stream selectors, reduce the time range of the query, or adjust parallelization settings

Or for un-shardable queries:

un-shardable query too large to execute on a single querier: (query: <size>, limit: <limit>); consider adding more specific stream selectors or reduce the time range of the query

Cause:

Even after query splitting and sharding, individual query shards exceed the per-querier byte limit.

Default configuration:

max_querier_bytes_read: 150GB (per querier)

Resolution:

Add more specific stream selectors.
Reduce the time range or Break large queries into smaller time ranges.
Simplify the query if possible - some queries cannot be sharded.

Increase the limit (requires more querier resources):

limits_config:
  max_querier_bytes_read: 200GB  # default is 150GB

Scale querier resources

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Interval value exceeds limit

Error message:

[interval] value exceeds limit

Cause:

The range vector interval (in brackets like [5m]) exceeds configured limits.

Resolution:

Reduce the range interval in your query:

# If [1d] is too large, try smaller intervals
rate({app="foo"}[1h])

Check your configuration for max_query_length limits. The default is 30d1h.

Properties:

Enforced by: Query Engine
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Time range errors

These errors relate to the time range specified in queries.

Error: Query time range exceeds limit

Error message:

the query time range exceeds the limit (query length: <duration>, limit: <limit>)

Cause:

The difference between the query’s start and end time exceeds the maximum allowed query length.

Default configuration:

max_query_length: 721h (30 days + 1 hour)

Resolution:

Reduce the query time range:

# Instead of querying 60 days
logcli query '{app="foo"}' --from="60d" --to="now"

# Query 30 days or less
logcli query '{app="foo"}' --from="30d" --to="now"

Increase the limit if storage retention supports it:
YAML
```
limits_config:
  max_query_length: 2160h  # 90 days
```

Properties:

Enforced by: Query Frontend/Querier
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Data is no longer available

Error message:

this data is no longer available, it is past now - max_query_lookback (<duration>)

Cause:

The entire query time range falls before the max_query_lookback limit. This happens when trying to query data older than the configured lookback period.

Default configuration:

max_query_lookback: 0 (The default value of 0 does not set a limit.)

Resolution:

Query more recent data within the lookback window.
Adjust the lookback limit if the data should be queryable:
YAML
```
limits_config:
  max_query_lookback: 8760h  # 1 year
```
Caution
The lookback limit should not exceed your retention period.

Properties:

Enforced by: Query Frontend/Querier
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Invalid query time range

Error message:

invalid query, through < from (<end> < <start>)

Cause:

The query end time is before the start time, which is invalid.

Resolution:

Swap start and end times if they were reversed.
Check timestamp formats to ensure times are correctly specified.

Properties:

Enforced by: Query Frontend/Querier
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Required labels errors

These errors occur when queries don’t meet configured label requirements.

Error: Missing required matchers

Error message:

stream selector is missing required matchers [<required_labels>], labels present in the query were [<present_labels>]

Cause:

The tenant is configured to require certain label matchers in all queries, but the query doesn’t include them.

Default configuration:

required_labels: [] (none required by default)

Resolution:

Check with your administrator about which labels are required.

Add the required labels to your query:

# If 'namespace' is required
{app="foo", namespace="production"}

Properties:

Enforced by: Query Frontend
Retryable: No (query must include required labels)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Not enough label matchers

Error message:

stream selector has less label matchers than required: (present: [<labels>], number_present: <count>, required_number_label_matchers: <required>)

Cause:

The tenant is configured to require a minimum number of label matchers, but the query has fewer.

Default configuration:

minimum_labels_number: 0 (no minimum by default)

Resolution:

Add more label matchers to meet the minimum requirement:

# If minimum is 2, add another selector
{app="foo", namespace="production"}

Properties:

Enforced by: Query Frontend
Retryable: No (query must meet requirements)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Timeout errors

Timeout errors occur when queries take too long to execute.

Error: Request timed out

Error message:

request timed out, decrease the duration of the request or add more label matchers (prefer exact match over regex match) to reduce the amount of data processed

Or:

context deadline exceeded

Cause:

The query exceeded the configured timeout. This can happen due to:

Large time ranges
High cardinality queries
Complex query expressions
Insufficient cluster resources
Network issues

Default configuration:

query_timeout: 1m
server.http_server_read_timeout: 30s
server.http_server_write_timeout: 30s

Resolution:

Reduce the time range of the query.

Add more specific filters to reduce data processing:

# Less specific (slower)
{namespace=~"prod.*"}

# More specific (faster)
{namespace="production"}

Prefer exact matchers over regex when possible.
Add line filters early in the pipeline:
logql
```
{app="foo"} |= "error" | json | level="error"
```

Increase timeout limits (if resources allow):

limits_config:
  query_timeout: 5m

server:
  http_server_read_timeout: 5m
  http_server_write_timeout: 5m

Use sampling for exploratory queries

 {job="app"} | line_format "{{__timestamp__}} {{.msg}}" | sample 0.1

Check for network issues between components.

Properties:

Enforced by: Query Frontend/Querier
Retryable: Yes (with modifications)
HTTP status: 504 Gateway Timeout
Configurable per tenant: Yes (query_timeout)

Error: Request cancelled by client

Error message:

the request was cancelled by the client

Cause:

The client closed the connection before receiving a response. This is typically caused by:

Client-side timeout
User navigating away in Grafana
Network interruption

Resolution:

Increase client timeout in Grafana or LogCLI.
Optimize the query to return faster.
Check network connectivity between client and Loki.

Properties:

Enforced by: Client
Retryable: Yes
HTTP status: 499 Client Closed Request
Configurable per tenant: No

Query blocked errors

These errors occur when queries are administratively blocked.

Error: Query blocked by policy

Error message:

query blocked by policy

Cause:

The query matches a configured block rule. Administrators create tenant policies and rate limiting rules to block specific queries or query patterns to protect the cluster from expensive or problematic queries.

Resolution:

Check with your Loki administrator about blocked queries and to review policy settings.
Modify the query to avoid the block pattern:
- Change the stream selectors
- Adjust the time range
- Use different aggregations
Request the block to be removed if the query is legitimate.

Configuration reference:

limits_config:
  blocked_queries:
    - pattern: ".*"          # Regex pattern to match
      regex: true
      types:                 # Query types to block
        - metric
        - filter
      hash: 0               # Or block specific query hash

Properties:

Enforced by: Query Engine
Retryable: No (unless block is removed)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Querying is disabled

Error message:

querying is disabled, please contact your Loki operator

Cause:

Query parallelism is set to 0, effectively disabling queries for the tenant.

Resolution:

Contact your Loki administrator to enable querying.

Check configuration for max_query_parallelism:

limits_config:
  max_query_parallelism: 32     #(the default)

Properties:

Enforced by: Query Frontend
Retryable: No (until configuration is fixed)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Multi variant queries disabled

Multi variant queries are an experimental feature that enables support for running multiple query variants over the same underlying data. For example, running both a rate() and count_over_time() query over the same range selector.

Error message:

multi variant queries are disabled for this instance

Cause:

The query uses the variants feature, but it’s disabled for the tenant or instance.

Resolution:

Remove variant expressions from the query.

Enable the feature if needed:

limits_config:
  enable_multi_variant_queries: true  #default is false

Properties:

Enforced by: Query Engine
Retryable: No (until feature is enabled)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Pipeline processing errors

Pipeline errors occur during log line processing but don’t cause query failures. Instead, affected log lines are annotated with error labels.

Understanding pipeline errors

When a pipeline stage fails (for example, parsing JSON that isn’t valid JSON), Loki:

Does NOT filter out the log line
Adds an __error__ label with the error type
Optionally adds __error_details__ with more information
Passes the log line to the next pipeline stage

Error types

Error Label Value	Cause
`JSONParserErr`	Log line is not valid JSON
`LogfmtParserErr`	Log line is not valid logfmt
`SampleExtractionErr`	Failed to extract numeric value for metrics
`LabelFilterErr`	Label filter operation failed
`TemplateFormatErr`	Template formatting failed

Viewing pipeline errors

To see logs with errors:

{app="foo"} | json | __error__!=""

To see error details:

{app="foo"} | json | __error__!="" | line_format "Error: {{.__error__}} - {{.__error_details__}}"

Filtering out errors

To exclude logs with parsing errors:

{app="foo"} | json | __error__=""

To exclude specific error types:

{app="foo"} | json | __error__!="JSONParserErr"

Dropping error labels

To remove error labels from results:

{app="foo"} | json | drop __error__, __error_details__

Authentication and connection errors

These errors occur when connecting to Loki, often when using LogCLI.

Error: No org ID

Error message:

no org id

Cause:

Multi-tenancy is enabled but no tenant ID was provided in the request.

Resolution:

Add the X-Scope-OrgID header in your request.

For LogCLI, use the --org-id flag:

logcli query '{app="foo"}' --org-id="my-tenant"

In Grafana, configure the tenant ID in the data source settings.

Properties:

Enforced by: Loki API
Retryable: Yes (with tenant ID)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Authentication configuration conflict

Error message:

at most one of HTTP basic auth (username/password), bearer-token & bearer-token-file is allowed to be configured

Or:

at most one of the options bearer-token & bearer-token-file is allowed to be configured

Cause:

Multiple authentication methods are configured simultaneously in LogCLI.

Resolution:

Use only one authentication method:

# Basic auth
logcli query '{app="foo"}' --username="user" --password="pass"

# OR bearer token
logcli query '{app="foo"}' --bearer-token="token"

# OR bearer token file
logcli query '{app="foo"}' --bearer-token-file="/path/to/token"

Properties:

Enforced by: LogCLI
Retryable: Yes (with correct configuration)
HTTP status: N/A (client-side error)
Configurable per tenant: No

Error: Run out of attempts while querying

Error message:

run out of attempts while querying the server

Cause:

LogCLI exhausted all retry attempts when trying to reach Loki. This usually indicates:

Network connectivity issues
Server unavailability
Authentication failures

Resolution:

Check Loki server availability.
Verify network connectivity.
Check authentication credentials.
Increase retries if transient issues are expected:
Bash
```
logcli query '{app="foo"}' --retries=5
```

Properties:

Enforced by: LogCLI
Retryable: Yes (automatic retries exhausted)
HTTP status: Varies
Configurable per tenant: No

Error: WebSocket connection closed unexpectedly

Error message:

websocket: close 1006 (abnormal closure): unexpected EOF

Cause:

When tailing logs, the WebSocket connection was closed unexpectedly. This can happen if:

The querier handling the tail request stopped
Network interruption occurred
Server-side timeout

Resolution:

LogCLI will automatically attempt to reconnect, up to 5 times.
Check Loki querier health if reconnections fail.
Review network stability between client and server.

Properties:

Enforced by: Network/Server
Retryable: Yes (automatic reconnection)
HTTP status: N/A (WebSocket error)
Configurable per tenant: No

Data availability errors

These errors occur when requested data is not available.

Error: No data found

Error message:

no data found

Or an empty result set with no error message.

Cause:

The query time range contains no matching log data. This can happen if:

No logs match the stream selectors
The time range is outside the data retention period
Log ingestion is not working
Stream labels don’t match any existing streams

Resolution:

Verify the time range contains data for your streams.

Check if log ingestion is working correctly:

# Check if any data is being ingested
logcli query '{job=~".+"}'

Verify stream selectors match existing log streams:

# List available streams
curl http://loki:3100/loki/api/v1/series

Check data retention settings to ensure logs are still available.
Use broader selectors to test if any data exists:
logql
```
{job=~".+"}
```

Properties:

Enforced by: Query Engine
Retryable: Yes (with different parameters)
HTTP status: 200 OK (with empty result)
Configurable per tenant: No

Error: Index not ready

Error message:

index not ready

Or:

index gateway not ready for time range

Cause:

The index for the requested time range is not yet available for querying. This can happen when:

Index files are still being synced from storage
The index gateway is still starting up
Querying data older than the configured ready index period

Default configuration:

query_ready_index_num_days: 0 (all indexes are considered ready)

Resolution:

Wait for the index to become available - this is often a temporary issue during startup.
Query more recent data that’s available in ingesters:
logql
```
{app="foo"} # Query last few hours instead of older data
```

Check the configuration for index readiness:

query_range:
  query_ready_index_num_days: 7  #default is 0

Verify index synchronization is working correctly by checking ingester and index gateway logs.

Properties:

Enforced by: Index Gateway/Querier
Retryable: Yes (wait and retry)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Tenant limits

Error message:

max concurrent tail requests limit exceeded, count > limit (10 > 5)

Cause:

The tenant has exceeded the maximum number of concurrent streaming (tail) requests. This limit protects the cluster from excessive resource consumption by real-time log streaming.

Default configuration:

max_concurrent_tail_requests: 10

Resolution:

Reduce the number of concurrent tail/streaming queries.

Use batch queries instead of real-time streaming where possible:

# Instead of tailing in real-time
# Use periodic range queries
{app="foo"} |= "error"

Increase the limit if more concurrent tails are needed:

limits_config:
  max_concurrent_tail_requests: 20  #default is 10

Properties:

Enforced by: Querier
Retryable: Yes (when connections are available)
HTTP status: 429 Too Many Requests
Configurable per tenant: Yes

Storage errors

These errors occur when Loki cannot read data from storage.

Error: Failed to load chunk

Error message:

failed to load chunk '<chunk_key>'

Cause:

Loki couldn’t retrieve a chunk from object storage. Possible causes:

Chunk was deleted or moved
Storage permissions issue
Network connectivity to storage
Storage service unavailable

Resolution:

Check storage connectivity from Loki components.
Verify storage credentials and permissions.
Check for chunk corruption or deletion.
Review storage service status.

Properties:

Enforced by: Storage Client
Retryable: Yes (automatically)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: Object not found in storage

Error message:

object not found in storage

Cause:

The requested chunk or object doesn’t exist in storage. This might happen if:

Data was deleted due to retention
Compaction removed the chunk
Chunk was never written successfully

Resolution:

Check if data is within retention period.
Verify data was ingested successfully.
Review compaction jobs for issues.

Properties:

Enforced by: Storage Client
Retryable: No (data doesn’t exist)
HTTP status: 404 or 500 depending on context
Configurable per tenant: No

Error: Failed to decode chunk

Error message:

failed to decode chunk '<chunk_key>' for tenant '<tenant>': <error>

Cause:

A chunk was retrieved from storage but couldn’t be decoded. This indicates chunk corruption.

Resolution:

Report to Loki administrators for investigation.
Check for storage data integrity issues.
Note that the corrupted chunk data may be unrecoverable.

Properties:

Enforced by: Storage Client
Retryable: No (chunk is corrupted)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Troubleshooting workflow

Follow this workflow when investigating query issues:

Check the error message - Identify which category of error you’re encountering.
Review query syntax - Use the LogQL documentation to validate your query.
Check query statistics - In Grafana, enable “Query Inspector” to see:
- Bytes processed
- Number of chunks scanned
- Execution time breakdown

Simplify the query - Start with a basic selector and add complexity:

# Start simple
{app="foo"}

# Add filters
{app="foo"} |= "error"

# Add parsing
{app="foo"} |= "error" | json

# Add label filters
{app="foo"} |= "error" | json | level="error"

Check metrics for query performance:

# Query latency
histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, route))

# Query errors
sum by (status_code) (rate(loki_request_duration_seconds_count[5m]))

Review Loki logs for detailed error information:

kubectl logs -l app=loki-read --tail=100 | grep -i error

Test with LogCLI for more detailed output:
Bash
```
logcli query '{app="foo"}' --stats --limit=10
```

Learn more about LogQL Query Language
Configure appropriate query limits
Learn more about Query performance tuning
Review the LogCLI documentation
Learn more about LogQL query optimization
Review query performance best practices
Use query debugging features to analyze slow queries
Explore the Grafana Loki GitHub repository for community support