Grafana Labs blog on Grafana Labs

'Grafana's Big Tent' podcast: Anthropic on agentic coding, observability, and the future of software engineering

Grafana Labs Team — Fri, 10 Jul 2026 18:34:08

In this episode of "Grafana's Big Tent" podcast, hosts Mat Ryer, Senior Director of AI at Grafana Labs, and Tom Wilkie, CTO at Grafana Labs, sit down with Eric Burns, Field Executive Architect at Anthropic, to talk about why Anthropic bet early on running across every major cloud, what it's like watching a technology go from "interesting" to "obviously inevitable" in real time, and how agentic coding tools have changed the day-to-day of building software at Grafana Labs.

You can watch the full episode in the YouTube video below, or listen on Spotify or Apple Podcasts.

Note: The following are highlights from episode 9, season 3 of "Grafana's Big Tent" podcast. The transcript below has been edited for length and clarity.

Why Anthropic bet on being everywhere, not just one cloud

Tom Wilkie: We're a pretty big fan of Anthropic models. We do use models from other vendors. I've heard they exist. But I think one of the things that attracted us to the Anthropic models was the availability. We run across all the major cloud providers, across Microsoft and Amazon and Google and so on, and the fact that you can get your models in-region across all of them was a big win for us. How did that come about, or why is that Anthropic's strategy versus, I know Anthropic and OpenAI and Google have very different strategies for model availability.

Eric Burns: Yeah. I've only been here coming up on two years, which makes me near old guard at Anthropic, but not there for the early days in the origin story. It strikes me as kind of a classic second-mover situation where there was an early breakaway leader and they had kind of a slightly more vertically integrated, or at least single-partner, strategy. When you're trying to get your feet under you as a startup, you try to figure out where your opportunities are and you maximize what you can do within the opportunity space.

So I did a startup before this. One of my favorite quotes from that time was something like, "Great architecture is all in the constraints." If you're a second-mover lab, one of your constraints is that you need to line up a hosting partner and you need to tell a good story for that. I think it's a very significant maturity transition for a company to be able to work with many different partners and many different platforms. It's kind of ripping the Band-Aid at a technical level that you can put in all of this work before the first dollar of partner revenue arrives.

That's a substantial opportunity cost, and it might fail, and you might be left without a good result. But I think one of the things that makes Anthropic is the decision to pay that cost early and to generalize and be able to work across clouds. You can see that now in being the first model provider to reach all three major hyperscaler platforms. I think one of our core competencies is making our models run on a diverse set of hardware.

Natural language as the new UI for dashboards

Eric: One of the most fun things about working at Anthropic is that there's just this continuous progression of goosebumps moments where you're like, "Oh my gosh, I can't believe computers can do this." And also, "I can't believe I'm getting to see this up close in the lab as it's materializing."

One of those moments that I'll never forget was the first time that I saw that we had a model that was able to basically start with natural language and create streaming UX, just build charts at the speed of thought, build very capable React single-page apps in a single prompt. I thought, "Oh my gosh, this is going to completely transform how we deliver user experiences."

This has progressed into this idea that there's a UI paradigm that you can actually build and it's a legitimate choice now. Which is, you start with natural language and end with streaming charting and BI and structured data presentation. I feel ridiculous saying this out loud on a Grafana podcast, but it just does seem like the wind is at the back of that particular model, that UX design model right now.

Tom: One of the longest open issues on the Grafana repo was, how do I batch-edit things? Because from a UX perspective, actually doing good batch-editing UX is really hard, it turns out. I now just believe natural language is the best UX for doing certain tasks, like batch-editing dashboards.

Eric: Yes, absolutely.

Tom: In one of the early demos that Mat gave me for the Assistant, which I thought was very impressive because I understand how hard it was to achieve, was to make all the themes for the panels purple and change all the panels. Obviously, you show that to a salesperson and they're like, "Why do I want that?" But for an engineer, it's like, "Oh wow, that's really difficult to pull off."

And suddenly, there's a lot of pre-AI dashboard slop around, but now the AI is actually better at producing dashboards, we've found, than humans. It labels the axes. It puts reasonable names on them and titles.

AI adoption across Grafana Labs' engineering org

Tom: We obviously have all witnessed the revolution in software engineering. I don't think, at Grafana Labs, any of our engineers are writing code anymore. Our penetration for things like Cursor, Claude Code, Codex, and so on is basically now at 100% in our engineering teams. But I guess this is an observability podcast and we should probably talk about observability at some point. As you go and talk to customers and execs and the community, what are you seeing in terms of the agentic use cases in observability?

Eric: It's a very complex set of actors that are gradually reaching consensus. If I go back to what's happening with coding agents, I've seen this real shift at the executive discussion level over the last six months especially. Six months ago it was: Are the models going to get good enough? Are the coding agents actually going to get there? Is this something that we should inflict on our team, or can we kind of hang out in shadow IT land and some people come in and use it?

Now it has definitively shifted... If you equip all of your frontline engineers with a fire hose of output, all of your other systems are going to crumble under the amount of stress that they're putting on them. Now it's moving to a second-order problem of building heavy automated test coverage, which of course coding agents are great for, obviously having evals around anything that is non-deterministic, and basically stacking the way up to being able to deploy coding agents with some confidence.

Then the second realization, as orgs sort of get the baseline of internalizing officially blessed coding agents, whatever flavor they pick, is this realization that integration just got really, really cheap. Being able to take one internal system and string it to another one became a prompt and a one-shot and some smoke testing, as opposed to, if you're outsourcing it, massive spec document, throw it over the wall, several iterations, months later, you get this dashboard that has been scoped down to the shadow of what you were hoping it would be, and you're like, "I guess I'll do another turn."

The ability to wire all this stuff up just based on immediate need, and then just sort of chat with your data or build a dashboard based on some wild idea, I think this is putting even more pressure on strong, opinionated UI systems for delivering this stuff, and critically, ways to persist it. One of the uncomfortable flip sides of all this prolific output from product managers and people—who didn't think of themselves as coders, but they can write the requirements and they can vibe-code their way to stuff—is distribution and hosting is actually really hard.

If you've got these deeply rooted systems of record where all of your tribal knowledge and all of your state info and your telemetry lives, and you've got kind of a consistent way, for example Grafana, to deliver this to users, the integration is where the magic is happening. It is not just ripping the whole thing off and saying, "Well, anybody can have a database, and we can vibe-code all of the integration layers in between, so why would we need a rendering system?" People still need a way to onboard onto using a platform. It's like back to that old "Who Moved My Cheese?" [story]. If the UI is continuously shifting, you can't enable people on it. You can't train people, and you can't really document it.

Zigbee meshes and Home Assistant

Tom: I don't get to code that much. I've got a very large engineering team. I'm mostly a manager now, and I have a backlog of coding projects that I've wanted to do. I'm slowly ticking off that backlog.

This is the intersection of AI and observability, because I do a lot of home automation. Everything in my house is fully automated, and I really want to store a lot more telemetry about my Zigbee mesh, for instance. Sometimes you press a button and it doesn't quite work, or doesn't work as quickly as you would hope, and I want to know why. What do I need to optimize and fix?

I just opened this PR yesterday, actually, and it was mostly Claude Code. It's been on my backlog to go and instrument Zigbee2MQTT for a very long time. It's not hard. It's 500 lines of Node. I am not a JavaScript or TypeScript engineer. It would have taken me days, if not weeks, to have done that, because I would have had to learn a ton. But I can read it and it seems reasonable. It does the right thing. It's got 100% test coverage. The satisfaction of being able to tick off a bunch of personal projects has just been absolutely huge.

Eric: There are two really interesting principles—not principles, attributes of coding agents for certain personalities. One of them is, I've also gone down the rabbit hole of Home Assistant and linking everything to everything and automating it all. I use Grafana and my Home Assistant to look at my InfluxDB, which is all my sensor outputs and so on.

Tom: Terrific. A hot tip: if you use the Prometheus exporter in Home Assistant, I recently refactored all of that, again using Claude Code, two months ago, so it's now got loads more entities in Prometheus.

Eric: Excellent. I know, I'm turning into a serious Home Assistant nerd, so perhaps there's a separate conversation.

The 'recovering engineer' getting the dopamine hit back

Eric: This zone of things that I can trust Claude to solve is now a fairly complex software project, but the only upside: there's no business value generated. I can click something on my phone that I used to click on a wall panel, right? Things like that. Or I can see high-fidelity data that I couldn't see before. The cost-benefit is getting completely transformed in terms of the effort that I imagine something is going to take. It has been in steep log decay to the point where basically nothing feels out of reach in this sprawling home integration project.

There were certain things where I was like, "OK, well, I'm just going to grit my teeth and reach in there and write the code, or I'm just not going to do it at all." So the first property is, many things that were total wastes of time got so cheap that just dashing off a Claude prompt and then checking in an hour later and saying, "Oh my goodness, I got this thing." I wasn't expecting that pure upside.

The second one I think is really profound for managers, and I experienced this. The last time I wrote production code was five years ago. There's a half-life of the quality of your dev situation where I can step away for a month and come back, and it would be eight hours before I was back in the flow. You pull down the latest, you've got some break, somebody forgot to check in this other thing, there was a framework migration, now you have to go read the docs for this framework, just on and on. The ability to ramp back up into the flow is almost always instantaneous now, because Claude will just bash through whatever nonsense is keeping you from being in your dev loop.

Tom: We refer to managers in Grafana Labs as recovering engineers.

Eric: Yeah, right. It's the encapsulation of exactly that idea. Suddenly that recovering engineer can get the dopamine hit of solving a problem that they didn't used to be able to.

Do we still need to read the code?

Mat Ryer: Do you think we will end up in a situation where we've stopped looking at the code, like Assembly? We don't really look at Assembly unless we need to. Most people can't, though they probably can now thanks to Claude, etc. Do you think we'll get to the point where the code is like, you look at it if you need to debug something, otherwise you're good.

Eric: You know, if you talk to Boris [Cherny], the creator of Claude Code, his view is, "We're already there." About a year ago, I stopped dirtying my hands with reading the actual code. Now I operate at the pattern level.

Increasingly, lately, I've had this weird experience with Opus 4.6 where I'll think I see something smart and I'll interrupt it in a loop. Then it dawns on me that it's actually a step ahead of me, and I'm like, "Oh, I'm so sorry. I thought I understood that, but actually, I don't. You're already on the right track."

So there's a certain threshold where, you know, I fancy on a good day I'm a decent engineer and I kind of know what's going on. I had this very uncomfortable feeling of being in Claude's way as it was trying to solve the problem and benevolently deliver the thing that I was asking it for.

One tier of question is, should we go review the code? Another one entirely is, is a human, even a competent one that knows the code base, for some value of competence, adding value or reducing throughput? I think these are the questions that, back to the idea of software engineering collectively engaging with it, there's no one right answer. Again, it's a question of risk tolerance.

But I've definitely had this ratcheting sense of my value being pushed out of implementation in the same way that anybody that's ever written Assembly would feel that most likely a high-level language expressing programmer intent very well, and a strong performance and compilation stack, is going to outperform whatever any of us could do at the assembly level.

Mat: Yeah, I see that future. I really do. It's closer than we think. Very exciting.

"Grafana's Big Tent" podcast wants to hear from you. If you have a great story to share, want to join the conversation, or have any feedback, please contact the Big Tent team at bigtent@grafana.com.

Business intelligence plugins for Grafana: A support update

Thanos Karachalios — Thu, 09 Jul 2026 15:52:01

In January, we announced that Grafana Labs had assumed maintenance of the business intelligence (BI) plugins created by Volkov Labs, and committed to a six-month maintenance period.

Today, we’re sharing an update: we're extending our maintenance commitment through the end of 2026. As announced earlier this year, that commitment includes maintaining compatibility with recent Grafana releases while handling bug fixes, security updates, and community contributions on a best-effort basis.

We want to be transparent about why we’re extending the maintenance period, what we've learned over the past several months, and how we're thinking about support going forward.

A quick look back—and where things stand

In September 2025, our longtime partner Volkov Labs announced they had been acquired. In light of the news, we committed in January 2026 to taking over the maintenance and development of their popular BI plugin suite to ensure continuity for the Grafana community.

Since then, we've made significant progress on the priorities we outlined in January:

Compatibility: The plugins have been brought up to date with Grafana 13 and React 19—the core engineering work for the maintenance window is largely complete.
Bug fixes and community PRs: We've continued to triage issues and review community contributions on a best-effort basis.
Security: We've continued to monitor and respond to first- and third-party vulnerabilities.

What we've heard from you

Over the past several weeks we've been meeting directly with customers and solutions architects who rely on these plugins. The goal was simple: understand how and why you use them before we make any decisions about their future.

What we've learned has been genuinely valuable. Many organizations use plugins like Business Charts, Business Variable, and Business Text to go above and beyond what core Grafana panels can do, building highly specialized, persona-specific visualizations tailored to industry verticals, proprietary data, and even custom physical environments (think dashboards designed to be legible on large displays from across a room).

For these users, the value isn't just in the plugins themselves, but in their ability to customize panels to meet their unique needs.

Extending the support window

As noted above, we're extending the current maintenance commitment through the end of 2026. This gives us time to keep listening to how you use these plugins, and to evaluate the right long-term support for each one.

We'll share more concrete plans as they take shape, and communicate any changes proactively. No plugin will be removed or have support reduced without advance notice.

A note on custom code and support scope

Because several of these plugins let you write your own JavaScript, we want to set clear, honest expectations about what "supported" means.

Grafana Labs support covers the plugins themselves—their documented behavior, the stable APIs they expose, and bugs in the code we maintain. It does not extend to debugging user-provided code added to the editor of a panel plugin. User-provided code in the panel editors can break between Grafana versions, and we're not able to guarantee or troubleshoot it on your behalf.

As we firm up the roadmap, improving the documented extension points—so you can achieve the customization you need on a more stable foundation—is a key part of the conversation. We'll provide updates once those decisions are made.

How to reach us and share feedback

Grafana Cloud and Grafana Enterprise users: Please contact Grafana Labs support with any questions.
Grafana OSS users: Please use the community forums, the plugin GitHub repositories, or the Community Slack channels.

If these plugins are important to your workflows, we'd especially love to hear from you while the roadmap is still taking shape. Your input will directly influence where this goes next.

How to scale access control in Grafana Cloud

Jake Batty — Tue, 07 Jul 2026 17:02:52

One of the primary reasons organizations adopt Grafana Cloud is to create a single pane of glass across the data they collect from self-hosted systems, cloud providers, and third-party platforms. Bringing those signals together enables richer correlations, reduces tool sprawl, and makes it easier for teams to understand what's happening across their environment.

But as observability grows and becomes more centralized, access management becomes more important. When infrastructure metrics, application logs, business KPIs, and customer-specific data all live in the same platform, organizations need a scalable way to ensure the right people have access to the right resources without creating additional administrative overhead.

The good news is this can all be done directly in Grafana Cloud. To illustrate how this can work in practice, let's look at a fictional company modeled after a common scenario.

Meet AcmeCloud (and the challenge of manual provisioning access)

AcmeCloud, a platform-as-a-service provider, has set up a new Grafana Cloud instance to visualize infrastructure metrics, business metrics, sensitive application logs pertaining to each of their customers, and more—all in one place. Now they need to provision access across their organization.

They have hundreds of users that require access, and different roles have different needs:

Admins need full control of data and users
SREs need to configure data sources, explore telemetry, and build dashboards
Contractors, working on behalf of specific AcmeCloud customers, need to log in but only to see the dashboards and data relevant to their tenant

At this scale, manual provisioning isn't an option. Neither is trying to keep users in sync between AcmeCloud's identity provider and Grafana Cloud by hand. It's simply too time-consuming and too prone to errors.

For organizations facing similar growth, this is often the point where access management starts to become a scalability challenge rather than an administrative task.

Establishing a single source of truth with SSO and SCIM

Instead of handling everything manually, AcmeCloud relies on SSO (Single Sign-On) for authentication and SCIM (System for Cross-domain Identity Management) for provisioning.

SSO handles how users logged in, while SCIM ensures the right users and groups are automatically available in Grafana Cloud. There are many options for authenticating users in Grafana Cloud, including native integrations with identity providers as well as support for generic authentication methods. To see the full list of supported methods and integrations, read more here.

As users are added, removed, or updated in the identity provider, those changes are reflected in Grafana Cloud automatically. The same concept applies to groups, which are mapped directly to Grafana teams. Groups within the identity provider typically organize users by job function, which correlate to a Grafana team in most cases, since users working in the same role typically have the same use cases and permission levels.

The result is a single source of truth for identity, with no need to manually reconcile users or worry about configuration drift.

For organizations implementing this approach, Grafana's SSO and SCIM integrations make it possible to manage users and groups from the identity provider rather than treating Grafana as a separate identity system.

If you are new to the concept of SCIM provisioning, check out this blog to learn more.

Tip: Establish clear group naming conventions in your identity provider before enabling SCIM. It makes permission mapping significantly easier as your environment grows.

Building a layered access model with RBAC

With users now syncing into Grafana Cloud, the next step is defining what those users can actually do.

AcmeCloud opts to manage access primarily through role-based access control (RBAC).This helps to maintain scalability while providing greater flexibility and security through a combination of basic roles, team memberships, and resource-level permissions. Dive deeper on creating a RBAC strategy and configuration through the guidance in our documentation.

Basic roles in Grafana set the foundation, with assignments mapped from the identity provider:

Admins are assigned the Admin role
Internal AcmeCloud users from the SRE team receive the Editor role
Contractors receive no basic role at all

This allows the internal SRE team to access dashboards and data broadly. They can build dashboards and alerts, as well as query metrics, logs, and traces across all tenants. All that is missing now is the ability to configure or alter data sources.

Contractors can log in, but they're presented with an empty Grafana experience by default. For now, there are no available actions to take in Grafana, but they need the ability to view dashboards based on data solely containing their own tenant id.

To build on the permissions provided through basic roles, teams that were provisioned through SCIM are used to layer on additional access.

Users inherit permissions from both their basic role and any teams they belong to. This is an important distinction: permissions in Grafana are additive, not restrictive. A user's effective access is the combination of all assigned roles and team memberships.

For internal users from the SRE team, a shared team is granted additional capabilities, including the ability to configure and manage data sources. This extends their access beyond what the Editor role provides by default.

For organizations designing their own access model, a useful starting point is to assign the minimum role required and then use teams to grant additional capabilities as needed.

Tip: Avoid using highly permissive basic roles as a shortcut. Teams are often a more scalable way to manage access as responsibilities evolve.

Restricting dashboard visibility with teams and folders

Contractors are grouped into tenant-specific teams, with no additional capabilities added at that level. Instead, access is controlled through folder permissions.

Each team is given view access to a specific dashboard folder, so contractors can only see only the dashboards relevant to their tenant and nothing else.

Folder permissions provided a simple and scalable way to partition dashboard visibility without creating separate Grafana instances for every customer.

For AcmeCloud, this approach balanced operational simplicity with tenant isolation. Organizations managing multiple customers, business units, or environments often find folder permissions to be one of the simplest ways to segment dashboard access.

Tip: Design your folder structure with future growth in mind. Reorganizing hundreds of dashboards later can become a significant effort.

Enforcing data-level isolation with LBAC

Up to this point, AcmeCloud has been using RBAC to determine what users could access within Grafana Cloud. But dashboard visibility and permissions were only part of the problem.

AcmeCloud also needs to ensure that access to dashboards doesn't automatically mean access to all underlying data. This is where label-based access control (LBAC) came into play.

By using LBAC, data is scoped by the tenant. That way, even if a user has access to a dashboard, queries are restricted so they only return data associated with their assigned tenant. Read more here for guidance on configuration and limitations.

This adds a final layer of protection, ensuring that customer data remains isolated-even within a shared Grafana instance.

For organizations operating multi-tenant environments, this distinction is important. Dashboard permissions control what users can see, but data-level controls determine what data they can access. Both are required to build a complete access strategy.

Delivering the right experience for every user

From the contractor's perspective, the experience is intentionally limited.

After logging in via SSO, they're placed into their tenant-specific team and land on a predefined home dashboard. They can only view the dashboards within their assigned folder and they can't explore beyond that scope—no visibility into other tenants, and no risk of accessing unrelated data.

What made this setup work isn't any single feature. Instead, it's how all these features are combined to address scalability, flexibility, and security demands.

SCIM handles scale and lifecycle management. RBAC—implemented through basic roles, teams, and folder permissions—defines who can access resources. LBAC then ensures data-level isolation within those resources.

Individually, each of these capabilities is straightforward. Together, they form a layered access model that allows organizations to onboard large numbers of users, both internal and external, without sacrificing control.

Bringing it all together

As Grafana usage grows, access management becomes just as important as observability itself. Designing around user types, scope, and data boundaries from the start makes it possible to scale confidently without losing track of who can see and do what.

The AcmeCloud example demonstrates a pattern that many organizations can adopt: centralize identity with SSO and SCIM, use roles and teams to define capabilities, control visibility through folders, and enforce data isolation with LBAC.

A single pane of glass doesn't have to mean broad access to everything. With a layered approach to access control, organizations can centralize observability while still maintaining the security and governance required as their environments grow.

Full-stack observability in Grafana Cloud: How to investigate issues across services and infrastructure

Victor Padilla — Tue, 30 Jun 2026 16:00:56

Many times, the hardest part of troubleshooting isn’t fixing the actual problem. It’s figuring out where to start.

As engineers, it’s easy to lose count of how many times we’ve opened logs, then 10 metrics tabs, and another 10 tabs with trace queries, only to end up back in the logs trying to find a root cause. Modern applications run across several layers of services and infrastructure, and understanding an issue often means connecting information scattered across different resources, teams, and observability signals.

Grafana Cloud Application Observability and Kubernetes Monitoring bring that context together, providing a full-stack view across applications, infrastructure, and Kubernetes environments. Starting your investigation from a service, pod, node, namespace, or cluster, you can quickly jump to the logs, traces, and profiles that help explain what's happening, all within the workflows you already use.

In Grafana Cloud, this experience is powered by the knowledge graph, which automatically models your applications and infrastructure into a unified graph. This graph maps telemetry to each connected entity, including services, pods, nodes, clusters, databases, and cloud accounts. The resulting views help you visualize these relationships and observability data together in a single place, so you can correlate signals, understand dependencies, and move from symptom to root cause faster.

In this post, we'll walk through an example of how full-stack observability in Grafana Cloud helps you investigate issues across the application and infrastructure layers, and how you can customize knowledge graph configurations to fit your environment.

From entity to insights: a workflow example

Grafana Cloud includes multiple features and views for full-stack observability across your applications, infrastructure, and Kubernetes environments. This eliminates the need to manually write queries or jump between dashboards. By automatically bringing together signals and visualizing relationships between services and infrastructure, you can identify issues and find root causes faster.

Each of the following features is built for a different stage of investigation, with the entities, relationships, and insights behind them powered by the knowledge graph.

RCA workbench helps you investigate incidents by bringing insights, dependencies, and telemetry together in a single timeline.
Entity graph provides a visual representation of the relationships between services, infrastructure, and other components in your environment, making it easier to understand dependencies and identify potential root causes.
Entity catalog acts as a central inventory of all services and infrastructure discovered by the knowledge graph, combining health status, insights, metrics, and metadata so you can quickly identify what needs attention.

From any of these features, you can launch directly into logs, traces, and profiles for a given entity. The embedded Grafana Drilldown tab opens automatically with filters derived from entity configurations (more on that below), making it easy to correlate errors detected from metrics with other telemetry signals.

To illustrate how this works in practice, let's use an example. It's late in the evening and you've just received an alert through Grafana Alerting in Slack that makes you break out into a cold sweat. Unsure where to start, you follow the provided link to RCA Workbench, so you can explore all potential causes for a particular issue correlated over time and dependency for the impacted service. There are some insights about your failing service, so you go and take a look at the telemetry your application is emitting.

You've identified the symptom, but what's actually failing? Is the issue with the Kubernetes pod? Another service? The database? Good news: you don’t need to exit the workbench to check the whole picture.

Instead of jumping between tools, you can explore the service's connected entities, including microservices and its frontend component, directly from the workbench. Several related services show activity, but one stands out: your PostgreSQL database appears to be failing.

A quick jump into the database’s logs reveals the root cause. The database has too many simultaneous connections and is refusing new ones, which is causing some related services to break. From there, you can begin to troubleshoot, whether that's increasing resources or horizontally scaling additional instances.

You can also create and share shortened URLs that bring teammates directly to the same view, making it easier to collaborate during investigations.

You might notice that when you open one of the Drilldown views, some filters are already applied to surface only the most relevant data. This is because the knowledge graph is configurable and can be tailored to fit your needs. Let’s take a closer look at how these configurations work and how you can customize them for your environment.

Have it your way: customizing configurations

As shown in the example above, Drilldown is a powerful tool that helps you understand your data without learning an entirely new query language. However, pinpointing the fields and labels that are most useful across your environments can be a challenge, as every system and team may follow different conventions for structuring and emitting telemetry.

There are default configurations that control how the knowledge graph filters, narrows down, and correlates your observability data with the entities in your environment. These configurations cover common scenarios by mapping labels such as pod, namespace, and cluster, along with standard OpenTelemetry fields like service.name and service.namespace for logs.

There are many potential setups: different labeling strategies, OpenTelemetry or non-OpenTelemetry, internal conventions, and more. This results in an almost endless number of possible scenarios.

Instead of trying to support every configuration out of the box, we empower users to customize their own experience within the knowledge graph.

Creating and editing a configuration

You can create configurations for specific environments, apply them only to certain entity types, or define matchers based on entity properties. You can even configure them to query any base data source you choose.

Take the following example (shown in the GIF below), which shows a new configuration being created to map the entity property deployment.environment to the log label service_namespace, and the entity property service to the log label service_name. Furthermore, filters ensure this configuration is applied only to entities whose deployment environment starts with prod. This could represent a real scenario in which your production metrics use deployment_environment, while your logs only include service_namespace.

To learn more about creating and editing correlations, please check out our docs.

Resolving configuration conflicts

In some cases, configurations may overlap or conflict due to matching scenarios. When this happens, the priority order defined on the configuration page determines which configuration takes precedence.

Configurations are evaluated as an ordered list, so adjusting their priority allows you to control how conflicts are resolved.

Handling unmatched configurations

If no configuration matches (or if the mappings cannot be applied) you’ll be prompted with an additional screen that allows you to temporarily apply a configuration even if it doesn’t match automatically.

This ensures you can still explore telemetry signals without needing to immediately adjust your configuration.

Best practices for configurations

While the system is designed to be flexible, a few best practices can help ensure a smoother experience:

Create a sensible default configuration that matches most of your environments.
Add more specific configurations for special cases, such as different teams using different namespaces or environment labels for logs or traces, to achieve more granular telemetry filtering.
Place default configurations at the bottom of the priority list so more specific ones take precedence.
Use consistent fields and labels across metrics, logs, traces, and profiles to make correlation easier.
Add as many mappings as possible to narrow down searches. Mappings are optional, so if an entity property is missing, it simply won’t be applied as a filter.

You can also use the Grafana Terraform provider to automate the creation and management of configurations. To learn more, please check out our documentation for the knowledge graph and Terraform.

How to learn more

The knowledge graph in Grafana Cloud offers a powerful way to unify your observability signals and accelerate root cause analysis. To dive deeper into shaping your telemetry and optimizing your graph, explore the following resources:

Configure telemetry correlation: Learn how to define explicit mappings between entities and data sources in detail.
Instrumentation quality: Understand the baseline for shaping your traces and metrics to ensure your application data is correctly linked.
Sending OTLP data: Learn how to collect, process, and export telemetry data into the Grafana Cloud observability stack so you can check it directly on the entity views.

Grafana 13.1 release: observability as code updates, extending Grafana Assistant across more data sources, and more

Grafana Labs Team — Wed, 24 Jun 2026 21:11:19

Earlier this year, Grafana 13 laid the groundwork for making it easier and faster than ever to turn your data into actionable insights.

With our latest minor release, Grafana 13.1, we're building on that foundation, expanding observability as code, bringing Grafana Assistant to more data sources, and streamlining the everyday workflows teams rely on to visualize, analyze, and act on their data.

Below are just some of the highlights from Grafana 13.1. If you want to explore all the latest updates, please refer to the changelog or our What’s New documentation.

Managing dashboards as code: what's new in Git Sync

Git Sync, a feature that brings native GitOps workflows into your Grafana instance, reached general availability with the release of Grafana 13. We added features to give you more flexibility and control when managing your dashboards as code, including GitHub App authentication and support for GitLab, BitBucket, and pure Git.

But we didn’t stop there. Grafana 13.1 brings four more enhancements to Git Sync that make it even easier to incorporate observability as code into your day-to-day workflows.

Import dashboards straight into a provisioned folder

Generally available in all editions of Grafana

You can now import dashboard JSON straight into a Git Sync-provisioned folder, picking the file path, branch, commit message, and workflow as part of the import.

From a folder, hit Import and Grafana walks you through a provisioned import flow: pick the file path, branch, commit message, and workflow, and the dashboard is committed back to your repository as part of the import.

Uniqueness is path-based, so two dashboards can share a title as long as they live at different paths in the repo, and a conflicting path stops the import before anything is overwritten.

Sync dashboards at the root level

Generally available in all editions of Grafana

You can now sync dashboards at the root level, without a containing folder, so provisioned dashboards can live alongside your non-provisioned ones. This is useful when a repo represents your whole Grafana setup, or when forcing everything under one folder doesn’t align with how your team organizes dashboards.

Pick Sync external storage directly at root level without a containing folder in the setup wizard and your provisioned dashboards land at the root, alongside everything else, instead of being scoped under a single folder.

Make dashboard context visible by default

Available in public preview in all editions of Grafana

Git Sync-provisioned folders now render their README.md inline by default, so the context for a folder travels with it.

Just drop a README.md next to your dashboards in the repo and it shows up in Grafana, including links, ownership notes, runbooks, or whatever your team wants to see sitting alongside their dashboards.

Sign commits automatically

Generally available in in all editions of Grafana

Git Sync can now sign commits with GPG, SSH, or S/MIME keys, so your Git provider marks them as verified. This means teams with branch protection rules that require signed commits can now use Git Sync without friction. Until now, Git Sync could only create unsigned commits, which caused pushes to be rejected in those repositories.

To enable signing, configure a signing key on the repository, and Git Sync will automatically sign every commit it makes to that branch. If no signing key is configured, commits remain unsigned.

To learn more about Git Sync, check out our documentation.

Extending the reach of Grafana Assistant

We're continuing to expand where and how you can use Grafana Assistant, our AI-powered agent in Grafana Cloud. From connecting to more data sources across your stack to making Assistant easier to access in self-managed environments, these updates help bring AI-powered observability to wherever your data (and Grafana instance) lives.

Using Assistant with additional data sources

With Grafana 13.1, you can use Assistant to directly query eight additional Grafana data sources: Snowflake, Oracle, Elasticsearch, Dynatrace, Honeycomb, Zabbix, Jira, and MongoDB (shown below).

This makes it easier to ask a single question and get an answer that draws from across your observability stack, your databases, and your project-tracking tools—no context switching required. For example, an investigation that starts with an alert can pull in error rates from Dynatrace, query performance from Oracle and recent deployments from Jira, all in one conversation.

For each data source, Assistant queries your data using natural language, correlates signals across sources, and visualizes the results as Grafana dashboards.

Assistant now pre-installed in Grafana Enterprise

Grafana Assistant now comes pre-installed in Grafana Enterprise, with no plugin installation required. If you're a Grafana Enterprise user, you can connect your Grafana Cloud account to start using Assistant right away.

If you're a Grafana OSS user, you can still get access to Assistant by installing the plugin from the Grafana plugin catalog and connecting your Grafana Cloud account.

Faster, more flexible dashboarding

Grafana 13.1 brings a batch of improvements that make building and exploring dashboards faster and more flexible.

Section-level variables for rows and tabs

Generally available in all editions of Grafana

In Grafana 13, we introduced section-level variables, a feature that lets you apply variables to each row or tab in a dashboard, so you can reduce clutter and improve the overall organization of your dashboards. With the 13.1 release, these variable types are now generally available.

Traditionally, dashboard variables have applied to the whole dashboard at once: if you changed an $instance variable, for example, this would update every panel together. That was a big limitation when a single dashboard spanned more than one service, such an API gateway and a database. Teams would have to split services across separate dashboards just to give each its own filters, making it difficult to achieve a unified view of their data.

Section-level variables solve for this. Each row or tab can now carry its own independent variables, so an API gateway row can scope to one set of instances while a database row scopes to another, all in the same dashboard.

A revamped query editor

Available in public preview in all editions of Grafana

The improved query editor experience we introduced as private preview in Grafana 13, which makes complex panels easier to build and manage, is now in public preview, with two new capabilities:

Multi-select with bulk action: You no longer have to manage queries, expressions, and transformations one at a time. Instead, you can click Select… in the sidebar footer to enter multi-select mode. From there, you can check the specific items you want to work with. A new bulk actions bar also lets you delete, hide, or show several queries at once, switch the data source for multiple queries in a single step, or enable and disable transformations in bulk.
Stacked view: When you want to see the whole pipeline at once, the stacked view lays out all of your queries, expressions, and transformations in a single scrollable list.

Overall, these two new features combined with incremental improvements to the original release make creating and editing complex queries more straight-forward and faster than ever before.

More dashboarding and visualization updates

In addition to the new query editor and section-level variables, other data visualization updates in Grafana 13.1 include:

Quick filters and data grouping: The new Filter and Group by dashboard control combines filtering and grouping in one place, so exploring data is faster and more intuitive.
Series visibility through time series legends: The new Series visibility filter in the time series visualization lets you narrow visible series interactively, by name, label, or both, without touching the underlying query.
Enhancements to nested tables: Table panels with nested rows are now much more configurable, with cell styling and improvements to aggregation.
Copy and paste panel styles: In a couple clicks, you can replicate colors, line styles, and more from one panel to another.
Panel style presets (shown below): Apply curated colors, thresholds, and display options to time series, stat, gauge, bar gauge, and bar chart panels with a single click in the panel editor.

For more details, check out our What’s New and data visualization docs.

PDC support for more data sources

Generally available in Grafana Cloud

With Private Data Source Connect (PDC), you can create a private, encrypted tunnel between your Grafana Cloud stack and data sources running inside private networks, VPCs, or on-premises environments.

With Grafana 13.1, we’ve added PDC support for three new data sources: MQTT, GitHub, and IBM Db2.

With this update, you can connect Grafana Cloud to MQTT brokers for real-time IoT and sensor data; GitHub Enterprise Server instances for source control and project metrics; and IBM Db2 databases running on-premises or in a private cloud.

To get started, deploy a PDC agent inside your private network, then configure your data source in Grafana Cloud using its internal DNS name. To learn more, refer to the Private Data Source Connect learning path.

Learn more about Grafana

For an in-depth list of all the new features in Grafana, check out our Grafana documentation, the Grafana changelog, or our What's New documentation.

Join the Grafana Labs community

We invite you to engage with the Grafana Labs community forums. Share your experiences with the new features, discuss best practices, and explore creative ways to integrate these updates into your workflows. Your insights and use cases are invaluable in enriching the Grafana ecosystem.

Upgrade to Grafana 13.1

Download Grafana 13.1 today or experience all the latest features by signing up for Grafana Cloud, which offers an actually useful forever-free tier and plans for every use case. Sign up for a free Grafana Cloud account today.

Our Grafana upgrade guide also provides step-by-step instructions for those looking to upgrade from an earlier version to ensure a smooth transition.

Special thanks to our community

We extend our heartfelt gratitude to the Grafana community!

Your contributions, ranging from pull requests to valuable feedback, are crucial in continually enhancing Grafana. And your enthusiasm and dedication inspire us at Grafana Labs to persistently innovate and elevate the Grafana platform.

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!

Post-incident review for TanStack npm supply chain ransom incident: No unauthorized access to customer production systems

Joe McManus — Tue, 23 Jun 2026 20:24:27

On May 27, we completed our internal investigation of the recent TanStack supply chain ransom incident and confirmed our initial findings: The incident was strictly limited to Grafana Labs' GitHub environment. There was no unauthorized access to customer production systems, and the Grafana Cloud platform was not affected.

For an additional, independent audit, we engaged Mandiant, a leader in cybersecurity and incident response. We provided them with API access to Grafana Labs' log environment to conduct queries across our systems for their investigation, which started on June 1. Mandiant confirmed that there was “no evidence of code tampering or repository poisoning within public organizations or production repositories delivered to end users.”

Since we discovered the incident, the Grafana Labs security teams have been running two parallel workstreams: completing the investigation and hardening our security operations. We are publishing this blog in the spirit of transparency to share more details about our incident response and remediation efforts.

Summary and impact

If you’re looking for the short version instead of reading our previous updates, here is the TL;DR: The TanStack supply chain attack hit us on May 11 via the Mini Shai-Hulud campaign. At the time, we believed we had successfully rotated every credential involved in this incident. We missed one. I won’t blame this oversight on hubris; the data we had at the time simply led us to believe our rotation was exhaustive. We were mistaken.

A bad actor utilized that overlooked credential to clone our entire repository collection. They then reached out on May 16, demanding a ransom to prevent a code leak.

Since Grafana Labs is an open source company, you might wonder why this is a concern. While most of our source code is public, we do maintain private repos for things like internal tools and specific Grafana Cloud features. It was a heavy decision, but we stuck to our principles and the FBI’s documented guidance: We did not pay.

We launched our mitigation efforts immediately, and we confirmed that there was no unauthorized access to customer production systems, and the Grafana Cloud platform was not affected. We also confirmed that while our codebase was downloaded, it was not altered. Our customers and open source users do not need to take any action.

Grafana Labs’ response

We were alerted to the incident on a Saturday, and teams across the entire company took action quickly and decisively. (Or to borrow a phrase from one my favorite rappers Big Daddy Kane, ain't no half-stepping at Grafana Labs.)

In response, Grafana Labs suspended all GitHub applications on May 17, initiated a global code freeze on May 18, and conducted a cross-platform audit of Vault, GitHub, Okta, Kubernetes, AWS, GCP, and host logs to verify that no production customer data was compromised.

In the weeks following, our engineering teams contributed to a comprehensive audit that included but was not limited to:

Completing 1,500 security-focused PR reviews
Auditing 280 GitHub applications, stripping permissions and removing several
Scanning 1,200 repositories for any signs of tampering
Executing 2,300 PR reviews looking for unauthorized changes in a single critical repo
Finishing infrastructure audits and retiring legacy systems
Performing wide-ranging new access audits

It was a massive undertaking, but each team stepped up in an extraordinary way to do their part. Engineering, security, and cross-functional partners worked tirelessly to respond, demonstrating the collaboration and the shared commitment we have to our community and our customers that I have always valued here at Grafana Labs.

After the initial assessment, we found that in addition to source code, the downloaded content included GitHub repositories that some Grafana Labs teams use to collaborate on and store internal operational information and other details about our business. This includes, for example, business contact names and email addresses that would be exchanged in a professional setting and email addresses that were used in some past marketing campaigns. This was not information pulled from or processed through the use of production systems or the Grafana Cloud platform.

If you wish to know if email addresses with your domain were identified, please reach out to Grafana Labs support.

Incident timeline

All times are in UTC

19:21 11 May - First malicious code executed on self-hosted runners by Shai Hulud threat actors, leaking credentials. Rotated credentials.
07:21 14 May - First malicious commit made by the threat actor using grafana-delivery-bot, leaked from Shai Hulud attackers.
13:28 14 May - Data exfiltration of repos begins.
20:57 15 May - Data extortion threat actor publishes their extortion demand.
08:30 16 May - Grafana Labs security team becomes aware of the claimed ransom and begins seeking confirmation.
17:39 16 May - Compromise confirmed; incident declared.
19:33 16 May - All known affected credentials and GitHub applications suspended/rotated. Suspension and rotation of all other GitHub applications and accessible credentials begins.
21:10 16 May - Suspension of all GitHub applications completed.
16:40 17 May - All code changes made by GitHub application accounts associated with the threat actor identified and reverted.
16:52 17 May - Root cause, attack chain of compromise identified.
17:21 17 May - DockerHub credentials determined not compromised.
17:51 17 May - All malicious workflow runs identified. Final list of affected secrets compiled and rotated. Rotation of all other ci/common secrets from affected repos continues.
23:23 17 May - Last of the potentially accessible credentials confirmed rotated or suspended.
03:08 18 May - Begin freeze of all non-critical code and deployment changes.
08:00 25 May - All-engineering security hardening week commences.
10:58 26 May - Commit review completed, service thawing begins. A repository needs to have been fully reviewed and transitioned to use a GitHub application token broker for short-term, finely-scoped credentials before being thawed.
10:54 27 May - Transition from repos directly pushing images to DockerHub to pushing to Google Cloud Artifact Registry occurs.
27 May - Internal investigation complete. No additional attack activity or compromised credentials were discovered.
08:00 2 June - All-engineering security hardening week concludes.
20:43 3 June - Review of repositories for data loss completed.
18 June - Mandiant investigation completed, corroborating internal investigation.

What’s next

The investigation is now closed, but our work to improve security operations at Grafana Labs will continue. Dostoyevsky once noted that "when reason fails, the devil helps!" I’m quoting “Crime and Punishment” to underscore our philosophy: We only wanted to implement changes that actually moved the needle on security.

We’ve spent the past month executing high-impact controls, including a token broker, fine-grained access controls, additional alerting, and static analysis. In addition, we have moved off of certain GitHub Actions and now use more tightly scoped actions with short-lived tokens.

We have also started the process of compartmentalizing our GitHub organizations and isolating all archived repos into a dedicated organization with actions disabled.

We will share an overview of our response efforts and the technical details of how we improved our security posture from our post-incident review in the coming weeks.

ObservabilityCON 2026 is coming to San Francisco!

Grafana Labs Team — Thu, 11 Jun 2026 21:50:09

ObservabilityCON 2026 is heading to San Francisco, and even as the fog rolls in over the Golden Gate Bridge, the future of observability will never be clearer. Join us at our flagship observability event at Pier 27 from October 19–21, 2026, where the community will gather at the epicenter of tech to navigate the fast-moving AI era.

Whether you're building agentic workflows on your local machine or running them in production, you’ll get access to the latest innovations, thought leaders, and observability experts. This year's annual event will feature:

An opening keynote exploring what's new—and what's next—in the observability space, including a look at the latest Grafana Cloud features and AI-powered solutions.
Live demos of the newest agentic tools and workflows in Grafana Cloud.
One-on-one time with engineers, peer success stories, and a clear view of where observability is heading.
Hands-on workshops and technical deep-dives related to AI observability, digital experience monitoring, and more.
Tons of networking opportunities to connect with hundreds of your closest observability friends.

Stay tuned for more details on the full ObservabilityCON 2026 agenda coming in July.

How to register

Registration isn't open just yet, but you can sign up now for early access to be first in line for tickets and discount pricing. Limited tickets will be available at 50% off, so you'll want to secure your spot before they're gone.

Speaking and sponsorship opportunities

Have an observability story to tell? Whether your team reduced noise, cut costs, adopted OpenTelemetry, migrated to Grafana Cloud, built AI into your workflows, or overcame a major challenge, we'd love to put your story on the ObservabilityCON stage and have you share lessons learned and the big wins your team experienced.

The call for presentations (CFP) for ObservabilityCON 2026 speakers is now open and you have until July 1 to submit your story for our San Francisco agenda. Need a little inspiration first? Check out these on-demand talks from last year's ObservabilityCON event.

We also have limited sponsorship opportunities available for ObservabilityCON 2026. Please email sponsors@grafana.com to learn more.

Can’t make it to San Francisco?

Don’t worry: we're taking ObservabilityCON on a world tour.

ObservabilityCON on the Road is coming to four cities this fall: São Paulo, London, Madrid, and Bengaluru. This one-day event is packed with technical deep dives, hands-on AI demos, and an Ask the Experts booth.

Here are the dates for this year's stops:

São Paulo: November 4 (presented in Portuguese)
London: November 5
Madrid: November 24 (presented in Spanish)
Bengaluru: December 8

Plus we will be adding more cities to the line-up soon!

Preview AI-powered solutions, deepen your observability expertise, drive bigger business impact, and connect with Grafana Labs experts face to face. Find your event and sign up for early access.

The countdown to ObservabilityCON 2026 has officially begun! Sign up today so we can keep you posted on all the latest updates.

Automatically discover and remediate root causes with Grafana Assistant Investigations

Maurice Rochau — Tue, 09 Jun 2026 15:04:37

You can use Grafana Assistant Investigations to automatically discover incidents and help find root causes—and this AI-powered Grafana Cloud feature recently got a major upgrade to give you even more confidence in its findings.

You can read more about the behind-the-scenes effort in our new engineering blog Unprompted, where we get into harness engineering, context compaction, benchmarking, and keeping agents alive and working well in long-running sessions. In this post, we'll focus on how you can get the most out of this product iteration in conjunction with our other features.

Keep reading to learn about all the ways Assistant Investigations, currently in public preview, can help you improve your incident response so you can run your own “human on the loop” auto-remediation workflows at scale in Grafana Cloud.

Investigate anything with Assistant Investigations

At its core, Assistant Investigations is a highly sensitive and tuned background agent that’s capable of investigating anything within the observability space or developer lifecycle. It’s your problem finder and validator.

Find instrumentation gaps in your setup: Task Assistant Investigations with looking at your metrics, logs, traces, profiles, services, and labels, and correlate that with your code to spot any improvements for your setup.
Define degradation criteria and evaluate against them: Build a skill with Grafana Assistant that captures certain criteria you want to meet. Schedule AI-assisted investigations on top of them and get a report every day if you’re still within your operational parameters.
Use profiles to raise PRs to make your software faster: Kick off an AI-assisted investigation with the purpose of looking at profiles and improving latency in your login and registration service. Based on a correlation of telemetry data, you’ll get improvements posted to GitHub either as issue, PR, PR draft, or branch.
Analyze your user drop-off rate across services: As long as the data is available, Assistant Investigations can also look at business metrics for you. That way, you can correlate high latency with churn in your checkout process or timeouts on your payment provider with chargebacks. You connect the dots between business impact, value capture, and your telemetry.

Customize Assistant Investigations to fit your specific needs

Assistant Investigations also gives you full customization, just like Grafana Assistant. And it follows our "big tent" philosophy, so you can use the tools and agents you prefer. For example, you can:

Use skills with auto-approved tools in them that it will discover and use
Use MCP server integrations to wire up your entire stack to Grafana Cloud and give Assistant Investigations access to all necessary systems
Connect to code via GitHub and GitLab; bring in business data from Snowflake or Salesforce; look at feature flags in LaunchDarkly; or manage CI/CD with Jenkins.

Common use cases for Assistant Investigations

Next, let's look at some of the use cases that can make Assistant Investigations so valuable to your observability practice.

Multiplayer your problem

With Assistant Investigations, there’s no need to tackle problems alone. When your on-call colleague kicks off an investigation for a problem and pages another team, the other team can easily jump into the conversation. They can retrieve the investigation, steer it, ask follow-up questions, or provide valuable context.

With the Assistant workspace and its canvas, multiple users can put down relevant information so that whoever reviews the investigation during the incident or after it can see what’s going on.

Kick start your response by connecting to alerts and incidents

Incidents don't happen on your schedule, and every minute counts when you need to get your systems back online. With Assistant Investigations, you can easily integrate with Grafana Alerting and Grafana Cloud IRM to kick off an investigation when alerts fire or incidents are declared.

If you have GitHub, GitLab, or Cursor configured, you can also close the loop and raise PRs automatically.You can even configure Assistant to send Slack messages into a group channel to nudge colleagues to review the auto-fix.

With Alerting’s outgoing webhooks, you can go further down the customization route by sending a POST request to another agent that has access to Grafana Cloud via our hosted MCP server or gcx. This allows you to kick off any agentic workflow to act on alerts how you need to.

And the next time an alert fires and the investigation comes back with a false-positive, just ask Assistant to improve your alert.

Catch up on what changed overnight

Thanks to agentic coding, software moves faster than ever—and we’re here to move fast with you. With Assistant's automation capabilities, you can kick off structured investigations on a time basis and have the reports sent back to Slack to you personally or in a channel.

Don't let your users be the first one to tell you when something breaks. Even if you don't have the right alert set up, the next time your checkout service sees increased latency and churn, you’ll catch it faster. And if you set up a GitHub or GitLab integration, you can create the PR to fix it right from Slack

One piece of the larger Assistant ecosystem

Now let's look at a practical example that highlights how versatile the Assistant ecosystem is and where Assistant Investigations fits in. If you're not using Assistant already, you can read along to get a sense for what it can do. Or, if you're already using Assistant, you can follow these same steps in your environment.

To do so, we'll use the workspace feature, which lets you manage multiple Assistant conversations from a dedicated page.

Open Assistant and select Open in Workspace from the conversation menu.
Ask about a service of your choice. For the purposes of this example, I picked a demo payment service, but you can run this with any service you have enabled to support Assistant and Assistant Investigations.
Ask Assistant to explore your service and write a skill for it.

Congrats, you’ve customized Assistant with your first skill! Next time you talk about the payment service, the context from the skill gets pulled in.

Next, let's re-use that skill in Assistant Investigations.

Go to Assistant > Settings > Integrations > IRM webhooks and configure it for the incidents and alerts you want Assistant Investigation to run on.
Go to Assistant > Settings > Custom rules and add a rule scoped to Investigations that says to always search for skills when an AI-assisted investigation starts and to pick the service-relevant skills.

Whenever Assistant Investigation is triggered, you can be sure it follows your runbook now.

Note: You can set custom rules and skills to “Just me” or “Everybody.” With “Everybody,” everyone profits from your setup without having to set up anything themselves. Once configured, it applies to the whole stack.

Finally, let’s layer on some automations.

Ask Assistant to create an automation for you that uses your skill and runs every morning at 8 a.m.
Tell it to send you a Slack DM.

From now on, every morning at 8 a.m., your skill is executed and all out-of-the-ordinary findings are reported.

Note: In my example, I didn’t connect Slack on purpose to highlight how Assistant actively helps you configure certain parts of it as they become necessary.

This is just one example of all the ways you can use Assistant and Assistant Investigations. You'll get the best outcomes by playing around and using the different features in conjunction with each other, so start testing it out today!

How to generate real-world load tests using Grafana Cloud k6 and production telemetry

Matt Wimpelberg — Wed, 03 Jun 2026 16:14:15

For many development teams, a load test starts with a set of assumptions.

You pick 100 virtual users because it sounds reasonable. You ramp for 30 seconds because that's what the tutorial showed. You set a 500ms threshold because it feels like a good target. The test passes, you ship the release, and production falls over at 6 p.m. on a Tuesday because your synthetic load never resembled how real users interact with your application.

The good news is that, if you're running Grafana Cloud, you already have the data you need to run realistic load tests. Your dashboards capture how your users behave, including request rates, latency distributions, and traffic patterns over time. The signal is already there; you just need to connect it to your test configuration.

Using production telemetry to shape your tests makes the results far more meaningful. Instead of validating against a hypothetical workload, you're testing against patterns your systems already experience, leading to more reliable baselines, better thresholds, and fewer surprises in production.

This post walks through how to do exactly that: pull real telemetry signals from Grafana Cloud and translate them into a Grafana Cloud k6 testing scenario that reflects production environments (rather than assumptions).

VU count isn’t the only place to start

Virtual user (VU) count is an important part of how Grafana Cloud k6, the fully managed performance testing platform powered by k6 OSS, runs tests. However, VU count doesn’t always need to be the first lever you pull. When your goal is to test how a service behaves under a known request rate, arrival-rate executors let you express that intent more directly.

What you may care about is the arrival rate, or how many requests per second your system handles. VU count is a byproduct of arrival rate and response time. If your p95 latency doubles, you need twice as many VUs to sustain the same throughput. Start from VUs and you're tuning the wrong variable.

This distinction often comes down to API testing vs. website testing. There is typically a difference between trying to simulate a number of "real users," including their actions and pauses and how those translate into backend capacity, vs. simulating a certain throughput on API endpoints. While the underlying test is ultimately similar, how you derive those traffic levels depends on the type of telemetry data you're working with. For instance, analyzing and simulating full user sessions would align more with Grafana Cloud Frontend Observability, whereas the metrics described below are focused on deriving the traffic levels of your API endpoints.

k6 has two executor types that map directly to how real traffic works:

constant-arrival-rate: a fixed number of iterations per second, regardless of how long each one takes. Use this when you want to hold a steady request rate and observe what happens to latency under that pressure.
ramping-arrival-rate: arrival rate changes over time according to stages you define. Use this when your traffic has a shape: a morning ramp, an evening peak, a midday lull.

Both of these need real numbers to be useful. Here's how to get them.

Step 1: Find your actual request rate

Open Grafana Cloud and run this query against your Prometheus or Mimir data source. Swap in your service name:

sum(rate(http_server_requests_total{app="leaderboard-api"}[$__rate_interval]))

Look at this over a representative window. A full week is better than a single day. You're looking for two things:

Your baseline rate during normal operation (requests per second)
Your peak rate, which is the highest sustained load you've actually seen

That peak number is your load test target. Not a guess or just 2x your average, but the actual peak your system has handled (or needs to handle).

If your baseline is 120 req/s and your peak is 340 req/s, you now have real numbers to work with.

Step 2: Get your latency baseline

This becomes your threshold. Run this query to get p95 latency under normal production load:

histogram_quantile(
0.95,
sum(rate(http_server_request_duration_seconds_bucket{app="leaderboard-api"}[5m])) by (le)
)

This value becomes the check in your Grafana Cloud k6 script, for example: http_req_duration: ["p(95)<280"].

Whatever this returns, such as 280ms, that's your threshold. If your load test pushes p95 above 280ms, the test should fail.

Also pull p99 while you're here. A gap between p95 and p99 tells you something about tail latency behavior that will matter under load.

Step 3: Read your traffic shape

Effective traffic modeling isn't just about mimicking a 24-hour cycle; it's about modeling your test configuration after the specific performance question you want to answer. Pull your request rate panel and look at the line's shape, then choose one of these four customer-modeled patterns to guide your stages array:

Constant load: Stay at a small percentage above your baseline for an extended time to verify steady-state stability.
Stress: Step through increasing load levels to identify the exact breakdown point where latency or error rates spike.
Spike: Ramp to your observed peak quickly for a short duration to test how the system handles sudden bursts of traffic.
Endurance: Maintain a baseline load for a very long period to uncover resource leaks or performance degradation over time.

For example, a stress test designed to find the breaking point might use stages that step through increasing load levels:

stages: [
{ target: 60, duration: "5m" }, // 50% of baseline
{ target: 120, duration: "5m" }, // baseline
{ target: 240, duration: "5m" }, // 2x baseline
{ target: 340, duration: "5m" }, // observed peak
{ target: 0, duration: "5m" }, // ramp down and cool off
]

In this case, you're not making this up; you're reading it off the dashboard.

Step 4: Wire it all together in Grafana Cloud k6

Here's what a full scenario config looks like when built from real data instead of assumptions:

import http from "k6/http";
import { check } from "k6";
export const options = {
scenarios: {
realistic_load: {
executor: "ramping-arrival-rate",
// Derived from observed peak concurrency in Grafana
preAllocatedVUs: 50,
maxVUs: 200,
// Derived from your traffic shape panel
stages: [
{ target: 120, duration: "5m" }, // ramp to baseline (req/s)
{ target: 340, duration: "10m" }, // push to observed peak
{ target: 120, duration: "5m" }, // return to baseline
],
},
},
thresholds: {
// Derived from production p95 — not invented
http_req_duration: ["p(95)<280"],
// Derived from observed error rate in Grafana
http_req_failed: ["rate<0.01"],
},
};
export default function () {
const res = http.get("https://your-service.example.com/api/health");
check(res, { "status 200": (r) => r.status === 200 });
}

Every number in that config came from a dashboard query. When a teammate asks why you chose these values, you have an answer.

Step 5: Close the loop back in Grafana Cloud

Run the test and send results to Grafana Cloud k6. Now open the same Grafana Cloud dashboards you pulled the baseline from and add an annotation for the time the test ran.

You're now looking at the same panels for request rate, p95 latency, and error rate during a load test that was designed to match production conditions. If the test surfaces a problem, you already know which dashboard to go to. The debugging context is built in.

This is what the observability and testing loop is supposed to look like: production data informs the test design, and test results feed back into your observability stack. Neither side is isolated.

A simpler way: Generate your script with Grafana Assistant

While calculating all the values manually (request rate, p95 latency, stages) provides maximum control and fidelity, there is an easier path to achieving a production-ready script.

Grafana Assistant, the AI-powered agent in Grafana Cloud, includes a k6 script authoring capability that allows you to generate performance tests using natural language. This feature automates the heavy lifting of performance engineering by programmatically translating production-observed metrics—such as the exact request rates and latency thresholds identified in your panels—directly into a valid k6 configuration. Instead of manually parsing Prometheus queries and hard-coding values into a script, you can simply ask Assistant to derive a test from your service's existing traffic patterns.

Grafana Assistant analyzes your service context, including relevant metrics, logs, and traces, to mirror observed behaviors like morning ramps or evening peaks.

By automatically mapping these real-world traffic patterns to k6 arrival-rate executors and defined stages, Assistant significantly reduces manual effort while ensuring higher fidelity to production conditions. This ensures your tests aren't just syntactically correct, but are grounded in the actual pressure your systems face every day.

To learn more about k6 script authoring in Assistant, please check out our blog post and technical docs.

Wrapping up: what this all means for shift-left testing

Shift-left testing is an approach where testing happens earlier into the development pipeline to catch issues before they impact users. While you can apply this approach to load testing, it doesn’t really help you if you’re testing on assumptions.

Shifting left with fidelity, however, means your pre-production tests reflect the pressure that’s actually applied by production workloads. That requires real data, which Grafana Cloud gives you, and then k6 gives you the test framework to act on it.

Grafana Cloud is the easiest way to get started with k6 and performance testing. We have a generous forever-free tier and plans for every use case. Sign up for free now!

Tempo 3.0 release: a new architecture for scale and lower TCO, TraceQL metrics GA, and more

Tiffany Jernigan — Mon, 01 Jun 2026 17:59:54

Tempo started with a simple goal: make distributed tracing easier to run at scale. As tracing adoption has grown, however, so have the challenges, including higher data volumes, more complex architectures, and increasing demand for real-time insights directly from traces.

Over the last year, we’ve been evolving Tempo’s architecture to meet that moment. And today, we’re sharing the results of those efforts with the release of Tempo 3.0.

The latest major release of the open source distributed tracing backend introduces a Kafka-compatible architecture for microservices deployments that changes how Tempo ingests, stores, and queries trace data at scale.

With the general availability of TraceQL metrics, Tempo 3.0 also makes it easier to create metrics and gain insights directly from tracing data. Alongside improvements to query performance, trace redaction, and more, these updates help teams deploy tracing more efficiently and cost effectively.

You can continue reading and check out the video below to learn more about these and other updates. The Tempo 3.0 release notes and changelog also provide more in-depth details and include all the changes included in this release.

New Tempo architecture for lower TCO

In the Tempo 2.9 release blog and release notes, we introduced a new Tempo architecture, codenamed Project Rhythm, with three goals: remove the replication factor 3 (RF3) requirement, decouple the read and write paths, and lay the groundwork for lower total cost of ownership (TCO). In practice, this means storing one copy of your trace data instead of three, which directly reduces object storage costs.

With Tempo 3.0, that architecture is now the default.

Why we evolved the Tempo architecture

The previous Tempo architecture centered on ingesters. Distributors sent trace data to ingesters, ingesters buffered recent data and flushed blocks to object storage, and compactors handled backend block maintenance.

This created some challenges, including:

Tempo needed RF3 replication across ingesters for durability, and TraceQL metrics added a second set of blocks through the metrics-generator and local-blocks processor. This meant storing 3 to 3.5 times the actual volume of tracing data.
TraceQL metrics had availability limitations. Metric data was tied to individual generators, so if a generator went down, the data it held became temporarily unavailable.
The read and write paths shared the same components, so one bad workload could destabilize both. Real incidents made this clear: Mimir queries took down ingestion, slow queries starved fast ones, and large Tempo trace attributes OOMed queriers.

To learn more about these and other factors leading up to Tempo’s re-architecture, check out the GrafanaCON 2026 talk, “When theory meets scale: Operating Mimir and Tempo in production.”

These updates to Tempo also align with other recent architectural changes we’ve made to Mimir, Loki, and Pyroscope to optimize performance across the OSS projects.

Splitting the read and write path

The solution to these issues was to fully separate the read and write paths so neither can affect the other. As a result, one of the main architectural changes in Tempo 3.0 is that the read and write paths are now split while in microservices mode, reducing storage and overhead while improving performance and reliability.

The distributor still receives trace data first, but what happens next is different. In microservices mode, the distributor writes that data to a Kafka-compatible system, which becomes the durable buffer between ingestion and the rest of the system. From there, block-builders consume trace data and write Parquet blocks to object storage, while live-stores, a read-path component, consume the same stream to serve recent traces before those blocks are available in storage. This is what lets newly ingested traces show up in queries quickly, without tying recent reads to the old ingester path.

The metrics-generator now consumes trace data from the same Kafka topic (a named stream that multiple components can read from independently), instead of receiving it through the previous distributor push path. Compaction also moves into the new architecture: compactors are replaced by the backend scheduler and backend worker, which coordinate compaction, retention, blocklist maintenance, and redaction jobs.

For smaller deployments, monolithic mode is still the simpler option with no Kafka required. Everything runs in a single binary.

To learn more, check out our Tempo architecture documentation.

Deeper insights: TraceQL metrics is now GA

TraceQL metrics, now generally available, lets you query ad-hoc metrics directly from trace data, making it easy to answer questions about performance, error rates, and service behavior across distributed systems.

Using TraceQL metrics, you can compute time series directly from trace data using queries such as:

{} | rate() by (resource.service.name)

With the new architecture, TraceQL metrics no longer depends on the metrics-generator component and is enabled by default. Recent TraceQL metrics queries are served by the live-store, and historical TraceQL metrics queries read RF1 blocks from object storage.

This matters during migration: Tempo 3.0 can query old RF3 blocks for normal trace search and trace-by-ID lookups, but TraceQL metrics queries only read RF1 blocks. If your 2.x deployment didn’t use the local-blocks processor, TraceQL metrics results cover data ingested after the upgrade.

Note: Alerting on TraceQL metrics remains experimental.

Comparison operators in TraceQL metrics

TraceQL metrics queries now support comparison operators: >, <, >=, <=, =, and !=. This lets you filter metric query results directly, so you can focus on the series that cross a threshold instead of scanning every returned series manually.

For example, this query returns the per-second span rate for every service except tempo-all, grouped by service name:

{ resource.service.name != "tempo-all" } | rate() by (resource.service.name)

Previously, you'd get results back for every service and filter from there. Now you can add a threshold directly to the query. For example, to return only services where the rate exceeds 1 span per second:

{ resource.service.name != "tempo-all" } | rate() by (resource.service.name) > 1

See the PR and release notes for more information. To mirror the specific example above, you can run the single-binary example.

Enhancements to the metrics-generator

Metrics-generator is an optional Tempo component that derives metrics from ingested traces, including span metrics, service graphs, and host info metrics.

In the 3.0 release, we focused on making it easier to manage cardinality, filter generated metrics, and compensate for sampling.

Per-label cardinality limiting

High-cardinality labels can quickly cause a cardinality explosion, consume the active series budget, and overwhelm the downstream systems (e.g, Prometheus, Mimir, or Cortex) where you remote write metrics. A common example is a label like http.url, where every unique URL can create another set of series.

The new per-label limiter gives operators more control on the upper bound of the cardinality per label. You can set the per-tenant max_cardinality_per_label override, and when a label exceeds that threshold, only that label’s value is replaced with __cardinality_overflow__. The rest of the label set is preserved. The new per-label limiter runs before the existing max_active_series / max_active_entities limiters, which remain as the last-resort safety net. This gives you a way to limit one problematic label without dropping all the other useful dimensions.

New metrics also help estimate per-label cardinality demand. Cardinality demand is tracked using HyperLogLog sketches, so the new metrics that help estimate per-label cardinality demand are estimates rather than exact counts. If cardinality drops back below the threshold, the limiter recovers automatically once the label ages out.

Check out the PR, configuration docs, troubleshooting docs, and HyperLogLog blog for more information. Benchmarks can also be found in the PR.

DRAIN-based span_name sanitizer

Span names with dynamic values (for example: GET /users/123) can create an active series per value and quickly drive up cardinality. The DRAIN-based span name sanitizer clusters similar span name values and uses heuristics to replace data-like tokens (e.g. numbers, uuids, etc) with a placeholder, so those span name values are grouped under a stable placeholder pattern instead of exploding active series.

In practice, this reduces total metrics cardinality, active series, and keeps metrics-generator and downstream systems stable from the cardinality explosion due to dynamic values. It’s disabled by default and can be enabled with span_name_sanitization, including a dry_run mode to estimate impact before applying changes.

See the PR for more details.

More flexible filtering with a new `include_any` policy

Span metrics filters now support a new include_any policy, giving you more flexibility to express OR-style inclusion rules that were previously impossible to configure.

Previously, multiple include policies behaved like a logical AND. That worked for strict filtering, but it made some configurations hard or impossible to express.

With include_any, you can express OR-style logic while keeping the existing include behavior unchanged.

metrics_generator:
processor:
span_metrics:
filter_policies:
# Main rule: only public/server traffic.
- include:
match_type: strict
attributes:
- key: kind
value: SPAN_KIND_SERVER
# Exception: also keep one critical internal span.
- include_any:
match_type: strict
attributes:
- key: kind
value: SPAN_KIND_INTERNAL
- key: resource.service.name
value: auth-service
- key: name
value: check-token

See the PR and documentation for configuration and more information.

Sampling compensation with probability sampling state

For environments that use head-based sampling, generated metrics can undercount real traffic unless the processor knows how to scale back the metrics from the sampled spans.

Tempo has support for span_multiplier_key config to configure a span or resource attribute that has the sampling ratio and scales the metrics accordingly. For example, if spans have attribute X-SampleRatio=0.1 (10% sampling), setting span_multiplier_key: "X-SampleRatio" results in each span counting as 10 requests.

New in 3.0, we also added the enable_tracestate_span_multiplier config on span metrics and service graphs to provide an alternative approach that extracts the multiplier from the W3C tracestate header using the OpenTelemetry probability sampling threshold.

When enabled, enable_tracestate_span_multiplier takes priority over span_multiplier_key.

For more information, check out the PR and documentation for more information.

Trace redaction to protect sensitive data

Trace data can include sensitive data, including personally identifiable information (PII). This release includes new redaction capabilities to help operators remove trace data when needed.

The new tempo-cli redact command submits redaction jobs to the backend scheduler over gRPC. Operators provide a tenant and one or more trace IDs, and Tempo creates backend jobs to rewrite the affected blocks in object storage, removing the matching trace data.

tempo-cli redact \
--tenant=STRING \
--trace-id=TRACE-ID,... \

See the PR and the documentation for more information.

Improvements to query performance

Tempo 3.0 delivers major query performance improvements across both vParquet5 and TraceQL execution.

vParquet5

Tempo uses the Apache Parquet columnar block format to store traces, and TraceQL queries operate on traces stored in those Parquet blocks.

vParquet5, the latest iteration of the Parquet-based columnar block format in Tempo, now supports up to 20 dedicated string columns per scope (span, resource, and event), doubled from 10, and up to 5 integer columns. It also does this dynamically, to reduce overhead to only what you actually use.

Dedicated columns can improve query performance by storing commonly used or queried attributes in their own columns, to put like-data together for best efficiency, and targeted reads when you query them. The Tempo CLI analysis and suggestion commands were also updated to help determine the best attributes to dedicate for your data. It's not always the case that more is better, and your optimal configuration might not need to use all available columns.

Please see the PR for more details.

TraceQL AST optimization

Tempo now automatically rewrites supported repeated conditions on the same attribute into a faster internal array operation, speeding up query execution. For example, a query like { resource.service.name="frontend" || resource.service.name="backend" || resource.service.name="api" } can be rewritten internally before execution, with no changes required from users.

This is a breaking change. The implicit array matching semantics of the operators != and !~ have changed: != now means NOT IN, and !~ now means MATCH NONE. Regex operands must now be strings or string arrays; values that were previously accepted because they could be converted to regex strings may now be rejected. Review the upgrade notes before moving to Tempo 3.0 if you use complex TraceQL queries.

TraceQL AST (Abstract Syntax Tree) optimizations can be disabled by setting skip_ast_transformations=all in the query frontend config. You can opt out per query with the hint skip_ast_transformations=all.

See the PR #6353 and PR #7012 for more details, including limitations and breaking changes.

A new way to enable Tempo MCP server

In Tempo 2.9, we introduced MCP server support as an experimental way for LLMs and AI agents to query and analyze Tempo data through TraceQL and other endpoints. In addition to enabling the feature via configuration, you can now use a dedicated flag to enable it as well:

--query-frontend.mcp-server.enabled=true

See the PR and documentation for more details.

Migrating to Tempo 3.0

Because Tempo 3.0 includes major architectural changes, we created a migration guide to help you move your Tempo 2.x configurations to 3.0. Also, see Upgrade your Tempo installation for breaking changes and migration guidance before upgrading to Tempo 3.0.

We also added a new migration command to the Tempo CLI.

For monolithic deployments, we recommend using:

tempo-cli migrate config old-config.yaml > new-config.yaml

For microservices deployments, we recommend using:

tempo-cli migrate config \
--kafka-address= \
--kafka-topic= \
old-config.yaml > new-config.yaml

The tempo-cli migrate command:

Removes obsolete config sections: ingester, ingester_client, and compactor
Adds Kafka (microservices mode only)
Removes the local-blocks processor
Disables compaction during parallel migration

For step-by-step instructions and more details, refer to the migration guide and the PR.

How to learn more

To see the full list of improvements, bug fixes, and breaking changes in Tempo 3.0, refer to the release notes and changelog.

If you’re upgrading from Tempo 2.x, start with the migration guide and upgrade notes and review the breaking changes carefully, especially the removal of ingesters, compactors, the local-blocks processor, v2 block format, the OpenCensus receiver, and the scalable single binary target.

If you are interested in learning more about Tempo, please join us on the Grafana Labs Community Slack channel #tempo, post a question in our community forums, reach out on X (formerly Twitter), or join our monthly Tempo community call. See you there!