Adaptive Logs overview

Adaptive Logs is a cost optimization feature in Grafana Cloud that lets you identify and reduce unneeded log volume. Most companies generate more logs than they need. By not storing logs that aren’t valuable to you, you can significantly reduce the number of logs you need to look through to find what you are looking for. Adaptive Logs makes recommendations which identify a percentage of logs which can be dropped based on query usage within your Cloud Logs environment.

How Adaptive Logs works

Adaptive Logs analyzes individual log lines, categorizes them into groups of matching patterns, then tracks the number of occurrences of each pattern match. It then looks at all query behavior in your Grafana Cloud Logs instance, and identifies how often log lines matching those patterns have been queried. Finally, Adaptive Logs presents a percentage of log lines per pattern that can be dropped with minimal impact, based on how often they are being queried.

You can review, edit, and accept Adaptive Logs recommendations in the Grafana Cloud Cost Management portal.

Based on applied recommendations, logs will be dropped upon ingestion by Grafana Cloud. Logs are dropped server-side, after they have been sent, but before they are indexed and made available to query. You will not be billed by Grafana Labs for these dropped logs.

Adaptive Logs workflow

The general workflow for using Adaptive Logs is as follows:

Review recommendations.
Select from the recommendations provided. You may edit recommendations as needed.
Apply recommendations.

Review recommendations

To access the Adaptive Logs user interface, in Grafana Cloud navigate to Home > Administration > Cost management > Logs cost management.
Under Optimize, click Adaptive Logs. If you have already applied one or more recommendations, the Adaptive Logs screen shows the drop rate currently being applied to each recommendation.

Understand recommendations

Adaptive Logs must wait some time before it can collect enough query data to analyze and make recommendations. The default time period is 15 days. After Adaptive Logs has collected enough data to analyze, it evaluates the previous 15 days of query activity to make recommendations. After Adaptive Logs starts making recommendations, they are updated daily, and reflect any changes in your ingest and query behavior.

A recommendation includes the following information:

Query:Ingest - the ratio of log lines matching this pattern returned in all queries, to log lines ingested, over the previous 15 days. For example, if there are 100 lines matching a pattern, and each of these lines has been queried 5 times over the previous 15 days, the query to ingest ratio would be 5:1.
Volume - the volume of logs ingested for this pattern over the previous 15 days.
Current drop rate - the drop rate currently being applied to the recommendation. By default, this rate is 0.
Recommended drop rate - the drop rate the recommendation engine suggests applying to your logs.
Projected savings - projected savings over the following 15 day period with the recommended drop rate. This value is extrapolated from analyzing the previous 15 days if the Adaptive Logs recommendations been applied.
Pattern - the pattern being used to power the recommendation.

Adaptive Logs evaluates queries on a given data source regardless of where they originate. It ignores queries that are too broad or not typically used for observability needs. Recommendations for logs that can be dropped include logs that are not included in current query patterns. As query patterns change over time, recommendations are updated.

Understand patterns

A pattern is a representation of log lines that includes both fixed and variable content. Patterns are grouped by the parts of the logs that don’t change, that is, the fixed content.

Here’s an example of a pattern:

level=debug ts=<TIMESTAMP> caller=metrics.go:<NUM> traceID=<HEX> duration=<DURATION> <*>

Let’s look at each of the following pieces of this pattern.

level=debug: This is a constant string appearing in all log lines matching this pattern.
ts=<TIMESTAMP>: The ’ts=’ string is fixed, and the <TIMESTAMP> string is variable, but always a TIMESTAMP data type. Everything inside an angular bracket represents variable content. Possible types are:
- <BYTESIZE>
- <DURATION>
- <HEX>
- <IP>
- <NUM>
- <TIMESTAMP>
- <UUID>
caller=metrics.go:<NUM>: This is a mix of fixed and variable content.
<*>: This represents variable alphanumeric string content.

Understand recommended drop rates

A drop rate is the percentage of logs that you choose not to ingest into Grafana Cloud after they have been sent. An 80% drop rate means roughly 8 out of 10 log lines are not ingested. The drop rate is the mechanism that creates savings in your Cloud Logs spending.

Each log pattern has its own recommended drop rate based on its ingest volume and query behavior. You can manage each recommendation separately on the Adaptive Logs screen. The Recommended column shows the recommended percentage of log lines to drop. The Current(%) column shows the percentage of lines currently being dropped, or planned to be dropped if changes have not yet been applied.

Manage recommendations

The Adaptive Logs user interface displays a list of patterns and recommendations for a suggested drop rate for each pattern.

To apply one or more recommendations, click the leftmost checkbox icon for the desired pattern(s), then click Apply selected recommendations. Click Apply to confirm and apply recommendations.

To modify and apply a single recommendation, click the edit icon in the Actions column, and enter a value in the modal. Valid values for the percentage of logs to be dropped are between 0 and 100.

To see a breakdown of the services which produced log lines matching a particular pattern, click, the > icon for the row.

To see log lines that match the label selectors and pattern, click the Open in Explore icon for the pattern.

To view an estimate of how much volume will be reduced when you click Apply drop rates without actually applying your changes, click Preview.

Manage recommendations in bulk

The Adaptive Logs user interface provides several tools to enable working with recommendations in bulk.

To apply recommendations in bulk, click the checkbox for the desired pattern(s), or all patterns, then click Apply selected recommendations. Click Apply to confirm and apply recommendations.
To only allow a drop rate up to a certain percentage, select the recommendations for which you would like to modify the drop rate, then select Batch edit rates. In the dialog box, enter the desired drop rate. This lets you configure the maximum drop rate you’d like to accept for the Adaptive Logs recommendations. When configured to 100% (the default), clicking Apply causes Adaptive Logs to configure all Current field values to the Recommended drop rate. Configuring a smaller percentage will set a ceiling for the recommended drop rate and any unlocked pattern with a recommended drop rate that exceeds the maximum drop rate will be reduced to match the Maximum drop rate. As an example, suppose you are presented two recommendations, foo with a suggested drop rate of 20% and bar with a suggested drop rate of 50%. Setting the maximum drop rate to 30% would set the Current rate for foo to 20%, and the Current rate for bar to 30%.
The Filter by Service menu lets you view recommendations for a specific service, and matches both full and partial service names.
To lock a recommendation, click the padlock icon. Locking a recommendation prevents edits from being applied to a specific pattern, including bulk edits. For example, setting a Maximum drop value would not impact a locked field. You can both lock and unlock a recommendation at any time.
To undo all edits that have not yet been applied, click Revert.

Apply recommendations

To apply one or more selected recommendations, click Apply selected recommendations, then click Apply.
To apply all recommendations, click Apply all recommendations, then click Apply.