New in Adaptive Logs: user-facing temporary pauses, exemptions, and per-service recommendations

We launched Adaptive Logs last year to help you optimize your log volumes and costs in Grafana Cloud, and we’ve been hard at work ever since making improvements based on your feedback.

Over the past couple of months, we’ve delivered several new features to help reduce toil, apply recommendations with precision, and—what we’re most excited about—confidently optimize your log ingestion while still providing peace of mind to your end users!

Let’s take a quick look at how you can start putting the new functionality to use today as we continue to deliver on our mission to ensure that all the log lines you ingest are worthy of your attention. 

Give developers peace of mind—and an escape hatch—by pausing Adaptive Logs

Adaptive Logs identifies and drops your lowest-value log lines. It does this by grouping log lines into patterns, then making recommendations based on how often those patterns are queried. This approach has proven very successful in helping observability teams optimize their Grafana Cloud spend, but we also realize there can be limits to this approach, as developers may prefer to hold onto log lines just in case past activity isn’t a predictor for what lies ahead. 

Of course, holding onto more logs than you need creates additional toil when troubleshooting, and it’s certainly more costly. But the “what if…” scenario exists nevertheless. This can make for hard conversations when you try to apply Adaptive Logs recommendations, and get some pushback from your application teams that “really need” those logs that they barely (or never) query.

But with the new ability to create temporary exceptions in Adaptive Logs, you no longer have to make that tradeoff! Developers and observability departments alike can get what they want, as it gives them both the ability to selectively ingest certain logs for a short period of time while still preserving the cost and toil optimization benefits the rest of the time. And all the Grafana Cloud users need to do is click a “Pause Adaptive Logs” button.

How it works

If you want to walk your developers through this new pause functionality, imagine any of these scenarios:

• An incident has been declared affecting your service or application.
• An intermittent issue keeps happening with your service that you haven’t been able to pinpoint.
• You’re deploying an update to your API server, and you want to ensure all goes well with the rollout.
• Something strange is happening with your service. Your spidey sense is tingling, but you’re just not quite sure what it is.

In all these cases, it would be really helpful to have your full suite of logs so you can quickly get to the bottom of whatever is happening. And that’s what this new feature does: it creates temporary exemptions to keep all logs that match certain filters.

Simply navigate to your log stream of interest in Grafana Logs Drilldown, and click Pause Adaptive Logs.

Next, click Apply, and the logs from that stream will come in at a 100% ingest rate. Refresh Logs Drilldown, or rerun your query, and voila!

After one hour, this temporary pause will turn off, and Adaptive Logs will behave as normal. You can also turn it off manually at any time using the same button. We specifically disallow overly large streams from being chosen in order to prevent too many logs from being ingested, and give organizations the confidence that they will still derive the cost savings benefits they intend.

So the next time something seems amiss in your system, your Grafana Cloud users can pause Adaptive Logs, temporarily stopping it from dropping log lines. They’re able to get the detailed insights they need in that critical moment, and when the dust settles, those same log lines will be dropped again. All without a need to alter Grafana configuration.

Now, let’s have that “do you really need those logs” conversation again :) 

Try it out today

The ability to pause Adaptive Logs is available in public preview for all Grafana Cloud customers querying in their Cloud Grafana instance, for that stack’s Logs data source. In addition to Logs Drilldown, it is also available in Explore mode in Grafana. (Administrators can disable the feature by following this documentation.)

We’re excited to deliver this feature, which creates a best of both worlds scenario for observability departments and end users alike. We look forward to improving on it with your teams’ feedback, and we’re looking to introduce similar features in other Adaptive Telemetry products down the road.

Retain business-critical logs with exemptions

Organizations often need to retain certain groups of logs for reasons beyond what our system can infer. For example, you may have to manage compliance, audit, and other customer-specific reasons that Adaptive Logs isn’t privy to!

Following in the footsteps of Adaptive Metrics, we are pleased to announce exemptions for Adaptive Logs. It works by telling the Adaptive Logs engine: Hands off, never drop these logs.

To use exemptions, navigate to in the Adaptive Logs UI, then go to the Configuration tab and click Add New Exemption. Provide the valid stream selector for the logs you would like to retain; you also have the option to provide a reason so you can remember why you did this in the first place. You can add, modify, or delete exemptions at any time.

Exemptions are generally available to all Grafana Cloud customers.

Drop logs with pinpoint granularity using per-service drops

We’ve learned that many applications share common services, use common frameworks, or are written with common standards, and thus produce similar log lines—especially at large organizations. Yet having granular control over the numerous services producing logs, and empowering optimization decision making at the team level, is critical to ensuring the right balance between optimization and outcomes.

To assist, we’ve added the ability to apply individual drop rates on a per-service basis. 

Let’s look at how this can work with a pattern that is commonly produced by several services. In this example, suppose you are especially interested in preserving logs matching the pattern for the mimir/ruler-querier service in its entirety. Let’s also assume service has just begun producing these log lines. 

To apply a drop rate for a recommendation specifically, open the “Edit recommendation” modal. In the per-service drop rate overrides section, enter the desired drop rate(s) for the specific service(s); in this case, for the mimir/ruler-querier service, enter 0.

This feature allows your organization to benefit from Adaptive Logs while respecting the different needs of different services and teams. It is available today in public preview for all Grafana Cloud customers.

Start getting more from your logs today

Per-service drop rates, exemptions, and user-facing temporary exemptions ease the anxiety of dropping log lines that you might need for compliance reasons or other unforeseen future needs. This should make Grafana Cloud users more comfortable with putting Adaptive Logs to use, and we cannot wait to see how much more you’ll be able to save on your log volumes thanks to these features.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!