Needle-in-a-haystack searches

Logline is a secondary indexing technology for Grafana Loki that dramatically accelerates “needle-in-a-haystack” (NIAH) searches, which are queries that search large volumes of log data for rare, highly specific values.

A needle-in-a-haystack query searches a large volume of logs for a rare, specific value. These searches usually return only a small number of matching log lines, but can be expensive because Loki may need to scan many chunks to find them.

Logline works by building a compact probabilistic index from ingested log data. During query execution, the index identifies which chunks contain matching log lines. Loki skips chunks that the index rules out, which reduces bytes scanned and query latency.

Acceleration does not change query results. If the index lookup fails or takes too long, Loki runs the query without acceleration and returns the same results.

Note
Queries that scan less than 10 GiB of log data are not accelerated. These queries already complete quickly without index assistance.

Needle-in-a-haystack search is generally available (GA) and is rolling out to Grafana Cloud Logs customers during the fourth quarter of 2026.

For background on the technology, see the Logline acquisition blog post.

Use cases

Logline is useful when you need to quickly locate a specific, rare value across large log volumes, for example:

A transaction ID or trace ID across weeks of payment logs
A specific user ID or device ID in high-volume application logs
An intermittent error string across a large multi-service environment

Supported queries

Logline accelerates the following LogQL filter expressions:

Exact string matches: |= "value"
Regular expression matches: |~ "value" (case-sensitive and case-insensitive using (?i)). A regular expression is accelerated when it contains a literal run of 6 or more characters that every match must contain. For example, in |~ "request_id=[0-9a-f]+ status=timeout", the literals request_id= and status=timeout are used to accelerate the search.
Filters on stream labels or structured metadata, checked before any parser stage, for example | trace_id="40ca1eb8-3f45-4d28-9033-62469a064a46". The value must have 6 or more characters.
Filters on fields parsed with json, logfmt, regexp, or pattern, for example | json | transaction_id="txn-123456". The value must appear exactly as written in the original log line. Values that contain quotation marks, backslashes, or control characters are not accelerated.

If a query has more than one filter, only one filter needs to match a supported pattern for the query to be accelerated. The query engine applies the remaining filters normally.

Examples:

Search for an exact transaction ID in payment logs:

{service="payments"} |= "txn-123456"

Case-insensitive search for an error phrase:

{service="api"} |~ "(?i)timeout exception"

Case-sensitive literal regular expression search for a trace ID:

{app="frontend"} |~ "trace-id: 40ca1eb8-3f45-4d28-9033-62469a064a46"

Regular expression with a long literal value surrounded by wildcards:

{env="production"} |~ ".*90429581536108547623.*"

Search for a structured metadata or stream label value, checked before the log line is parsed:

{service="api"} | trace_id="40ca1eb8-3f45-4d28-9033-62469a064a46"

Search for a value in a field parsed from JSON:

{service="api"} | json | transaction_id="txn-123456"

Pipeline with an accelerated filter followed by an additional filter (the index accelerates |=; the query engine handles !=):

{cluster="prod"} |= "user-id-789" != "healthcheck"

Search across multiple services for a specific device ID:

{env="production"} |= "device-abc-9876"

Limitations

The following queries still run, but without acceleration:

Filter values with fewer than 6 characters. Short values appear too frequently in the index to provide meaningful narrowing:
logql
```
{service="api"} |= "12345"
```
Negative filters (!= and !~). The index only accelerates positive filters that confirm a value is present:
logql
```
{service="api"} != "timeout exception"
```
logql
```
{service="api"} !~ "(?i)connection refused"
```
Numeric, duration, and byte-size comparisons. The index works with text values, not numeric ranges:
logql
```
{service="api"} | status >= 500
```
IP address filters. These use a specialized matching function that the index does not support:
logql
```
{service="api"} |= ip("192.0.2.1")
```
Filters joined with or. Logline does not yet support accelerating joined filters.
logql
```
{service="api"} |= "timeout" or "unavailable"
```
Filters that come after a stage that rewrites the log line, such as line_format, decolorize, or unpack. Filters before these stages are still accelerated:
logql
```
{service="api"} | line_format "{{.msg}}" |= "timeout exception"
```
Filters on fields after a stage that changes or discards labels, such as label_format, keep, drop, or unpack:
logql
```
{service="api"} | json | keep msg | transaction_id="txn-123456"
```
Regular expressions with no literal run of 6 or more characters. The index needs a fixed substring to look up:
logql
```
{service="api"} |~ "[0-9]+"
```
logql
```
{service="api"} |~ "(error|warning)"
```
Metric queries. Logline does not yet support accelerating Metrics queries.
logql
```
count_over_time({env="production"} |= "device-def-5432")
```