Telemetry-driven development: How to gain confidence in your coding agents' behavior with gcx and Grafana MCP

You’re about to click "Merge" on a PR, but you feel more anxious about it than you used to. Why? 

You did everything properly, by today’s standards: You used Claude to create a plan, giving it context from a GitHub issue and some Slack threads. You iterated on it a few times; Claude says it has the full picture. It feels ready, so you get Claude Code to implement the plan. You run a review agent against the diff and get some of those comments addressed. You start reading the diff, but it's long. You try to build a mental model of the changes and how they fit in with the system. A few more cycles of feedback and questions with Claude, and things feel … OK. So you create a PR. The AI review comes back all clean, and you get a "LGTM - ship it" from your teammate.

So why do you still feel uneasy? Well, it's probably because you don’t know the code diff the same way you would if you’d typed it all out yourself. 

Before AI, we built our mental models of system changes as we wrote the changes, and we relied on those models to ensure the code we deployed behaved as expected. However, that all happens less often when AI is the one making the changes. Coding agents have increased the speed of delivery, but our speed of understanding is falling behind. We need a better way to maintain our confidence in our changes.

Thankfully, more data can help us make better decisions in this scenario. What you lose in knowledge or confidence from having an agent generate your code, you gain from seeing your system run and behave the way you expect. Good observability can help you do just that throughout your software’s lifecycle—including the build step.

In this blog, I'll show you how you can gain confidence in your code and your agents by using some of our AI tools.

Give your coding agent access to your telemetry

First, let's get you up to speed on the tools we offer to help you adapt to this new landscape. 

gcx, the Grafana Cloud CLI, and the Grafana MCP server can both connect your agents to your Grafana instance and let them interact with that telemetry data. 

• Grafana MCP server, which you can run yourself or use through the hosted MCP server for Grafana Cloud stacks, provides opinionated tools for a set of common use cases.
• gcx is a more flexible, less opinionated tool which agents can explore, and allows you to build your own workflows. gcx can be used on Grafana Cloud stacks or self-hosted OSS or Enterprise instances.  

Both tools are now generally available.

If you’re using Claude Code, you can easily install the Grafana MCP server as a plugin. We also offer the Grafana Assistant plugin, which provides guidance on interacting with Grafana Assistant. gcx comes with its own bundle of skills, which you can install with gcx agent skills install. These skills can teach your agent to use gcx effectively and efficiently for some common workflows.

Let your agents understand your system’s existing behavior

We built these tools to address the challenges that come from working with agents, because agents will give more useful results if they understand how your system actually runs: realistic request rates and sizes, which code paths are busy and which are not. This data will stop an agent from over-optimizing a code path that's rarely used, or enable it to generate more realistic synthetic data for a load test.

As well as accessing your telemetry, agents can use your SLO definitions, Synthetic Monitoring checks, and more to get additional context on how you operate your system, so they can plan changes that best suit the way your team works. SLO definitions will show agents the headline metrics for your system. Synthetic Monitoring checks will show the desired behavior from an external caller’s perspective.

Say you’re adding a new provider to your payments service. Suppose you already have a dashboard showing RED metrics for your payments service and you can see that the p95 request latency for the existing provider Swipe is two seconds. You will explain to your agent that you expect 30% of the traffic to be diverted to this new provider, and you expect the latency to be the same. 

Your agent can fetch real telemetry data from your production environments to ground these requirements in reality, which might influence the suggested implementation from the agent since it knows what requirements and limitations it needs to deal with. For example, it could find the current payment request rate for your systems to get an estimated request rate for the new request handler it will build. It will also know to add a built-in latency when it mocks the payment provider for unit and integration testing.

The result will be logic that behaves as you expect in an environment similar to your production environment, since the agent was guided by real data instead of making assumptions based on its training data.

Did you know you can continue Grafana Assistant conversations on the command line? This can be really useful if you want to get context from your conversation into your coding agent. In the Assistant sidebar or Assistant Workspace, click the three dots, then click ‘"Hand off Conversation" to see a gcx command to pull the transcript.

Telemetry-driven development

Unless your system is brand new, it's likely emitting telemetry, and you have some dashboards or SLOs set up to monitor its health. These existing data points can become part of the spec for your agent, to complement your written specs and test criteria. 

If you’re new to observability, or you’re not sure what metrics or telemetry you want in your system, or if the work to add them seems overwhelming, take a look at gcx’s bundled gcx-observability skill, which will guide you and your agent through instrumenting your codebase, and show you what other observability features you could use. Take a look at the skill with gcx agent skills get gcx-observability.

To illustrate this, let's continue with our payment provider example, where we have an existing dashboard showing RED metrics. You need to make sure you can detect issues with the new provider, so you tell your agent to update the dashboard to sum the metrics by provider. With gcx or the Grafana MCP server, your agent can read and update these dashboards. It can also write the updated dashboard spec to source control or push it back into Grafana for you. It can even inspect the queries used in the dashboards and find the places in the codebase to update the telemetry.

To make the best use of the telemetry when you’re building, your agent also needs to iterate on the changes until the desired behavior is observed. To see the data in those metrics, it needs to export telemetry from your local build. This kind of setup task is much easier with AI: Agents have been trained on many OSS observability products and know how to run an OpenTelemetry Collector and send that data to your Grafana stack for you (See our CTO Tom Wilkie’s post Why open source is the cheat code for AI).

If you want a fully local setup instead, tell your agent to use the grafana/otel-lgtm Docker image for an out-of-the-box LGTM stack. You can configure gcx to also interact with this local Grafana instance. In our example, you could pull the payments dashboard from production with gcx resources pull and push it into your local Grafana instance. Agents (with an assist from the gcx documentation that they read along the way) can also help tweak some data source settings so you have a dashboard and metrics to validate against.

As a result, you have a fully automated feedback loop from your telemetry, which the coding agent can use to iterate on changes until it reaches your desired state.

Generate some traffic

From there, maybe you need to simulate some payment traffic to see how it behaves. This is another example of the kind of task that used to be time consuming and difficult: without AI, you’d need to understand the shape of the traffic, and either hand-write some sample requests to cover the scenarios you could think of, or use some fuzzer to put some variety into your traffic. Then you’d need to go and understand k6’s syntax for creating test scripts, and maybe also figure out how to run k6 locally. This is at least a day’s work.

Now, with AI, you can ask an agent to generate that sample traffic, write tests scripts and even stand up k6 in a Docker container for you. It can access the production telemetry to know what shape and rate of requests is most useful for you. Then, you can watch how your system behaves under some load. This gets you closer to proving the system will work in real conditions.

k6 comes with its own bundle of agent skills, which you can install with k6 x agent init <target> (for example k6 x agent init claude-code, or --all to cover every supported tool). Like the rest of this workflow, these skills draw on your telemetry: they can teach your agent to author new tests from realistic traffic, work out whether a failing run was the test or the system under test, and analyze trends across runs to propose fixes or tighten thresholds before they become an issue in production.

When you can see your local builds run, and you can see that it's behaving and performing as you expect under some realistic load, you’ll have more confidence. You can build faster and still feel in control.

Improvement loops

Agents can also suggest improvements for you. Tell an agent to reduce your system’s memory footprint by 10% and it can query profiling data, find hotspots in the code, and create a draft PR explaining what it changed. From there, you can use your k6 scripts and telemetry setup to prove the change worked.

The Tempo squad at Grafana Labs is already using a pattern like this: They have a custom agentic harness that will iteratively conduct a performance analysis on a development environment that has some significant load—typically a few terabytes of data in its bucket.

For each analysis, it performs some queries against the dev environment to get a baseline. Then it checks the profiles for the deployment to find candidates for improvements. If it finds a candidate, It implements the change, deploys it, and reruns the same query to compare to the baseline. With that data, the team can decide whether they want to commit the change. 

Telemetry doesn’t give you the full picture for UIs, unfortunately. However, we have released Agentic Testing (experimental), which allows you to use natural language to describe a set of instructions and validations for an agent to perform against a publicly available web application. These tests can help detect regressions in your UI, and they are easier to define and better equipped to deal with UI changes, than traditional browser tests.

Regaining control

Instead of ending up with a half-understood PR and agent-generated description, you could now have a PR with links to dashboards showing your local build behaving as you expect under realistic load. 

For our example, you could show where you’ve added the new payment provider labels in the metrics and logs, and if you exported the local build’s telemetry to Grafana, add a link to your new dashboard showing metrics for the new provider at the time you were running the load test. This will show the reviewer that the payment handler dealt with 30% of the load just fine.

There is much more evidence to help convince yourself, and your PR reviewers, that the change will behave as you expect. This extra evidence will reduce the cognitive load for your reviewer, too.

Grafana Assistant is the easiest way to get started with metrics, logs, traces, dashboards, and more in Grafana Cloud. We have a generous forever-free tier and plans for every use case. Sign up for free now!