How to build a trust platform for your agent with Grafana Agent Observability

Observing fast-growing agentic workloads is no small feat, especially if you try to build your own monitoring stack or rely solely on tools built for a time before LLMs.

At Grafana Labs, we know this all too well. 

Grafana Assistant went from an internal hackathon project to generally available in a little over six months. And while it was great to see how eager our users were to adopt it, that interest came with plenty of growing pains as we managed scaling and complexity behind the scenes. Reviewing contributions to the prompt, tool set, and agent harness became a nightmare as changes piled on from multiple teams. Bespoke dashboards just weren’t cutting it. (You can read more about our experience in this post on Unprompted, our new engineering blog.)

If you've been building agents, you've probably faced some similar frustrations. The good news? You don't have to blindly walk through the same growing pains that we did. We built Agent Observability as an internal tool to address these challenges, and today we're making it generally available to Grafana Cloud users to help you watch your agents, test their behavior, and step in when something's off. 

But having the right tool is only the first step. This post will document guidance and best practices for building a robust trust and monitoring setup to monitor your own agents, based on the phases we went through building Assistant.

Phase 0: Early prototype and initial deployment

It's exciting to start with a fresh idea, pick an agent framework, and say the first “Hello, world!” to a new agent. In fact, you can get surprisingly far by adding tools and single-shot examples to the system prompt, then doing some simple Q&A testing.

After getting the agent to a place where it generally feels solid and helpful, you can run an initial benchmark or test suite with a barebones set of use cases the agent needs to address. But once it’s deployed to production for the first time and the first issue appears, the cracks start to show. A customer may reach out regarding a strange conversation or an internal user might message on Slack with a screenshot of a conversation that went off the rails. How do you begin to troubleshoot this?

The next step in development is making sure that these conversations can be found and triaged efficiently. Like traditional software monitoring, this is done through instrumentation of the agent’s code.

Phase 1: Monitoring live traffic

There are two main facets that are important to monitor when observing an agent. The first is traditional engineering metrics such as latency, cost, number of tokens, and errors. These metrics can give developers a sense of the general maintainability of the agent, identify cost anomalies, and do an initial round of improvements to the agent, such as reformatting the prompt to introduce prompt caching and save on tokens. When the finance department comes down to the engineering team and asks for the costs to maintain an agent, this step will provide the answer.

The second facet is the conversations themselves. You can surface a surprising amount of feedback by spot-checking conversations and seeing how the agent behaves without any deeper analysis. Reading over a handful of transcripts gives a great general sense of how the agent is working. Instrumenting the conversations also makes it easy to triage customer requests, find errant conversations, and resolve them.

To set up instrumentation and visibility into an agent’s conversations, it is as simple as writing a config that fits in with agent frameworks, through the Grafana Agent Observability SDK. The SDK ships with coding agent skills to help with instrumentation. gcx, Grafana's new CLI, also comes with skills to help get these phases set up (see agento11y-instrument for more details).

Great, now there is a way to respond to issues that are raised with the agent in production. However, spot checking and traditional engineering metrics can only get a team so far. The next step is surfacing anomalies automatically, for which there are some stronger methods available.

Phase 2: Evaluating baseline performance

Agents are inherently non-deterministic and heavily customized for a wide range of use cases. This combination makes it hard to get a sense of agent quality using traditional metrics. How do you trust that they are working correctly in production? 

To address this, a combination of deterministic checks (such as regex or JSON validation) and LLM-judge evaluators is recommended for monitoring agent quality over time. LLM-judge evaluators work by asking another LLM to take a look at the transcript, or pieces of the transcript such as an individual tool call, and score it on an output scale, typically boolean (pass/fail) or Likert (1-5).

Evaluator templates available out of the box in Grafana, such as PII identification and Toxicity, are useful to get started. While these general LLM-judge evaluators like Helpfulness can give you more information about an agent's performance than pure engineering metrics, they can be prone to unreliability. 

That’s where custom evaluators can help tailor the metrics to more accurately diagnose agent behavior. When creating an evaluator, configure the prompt and the context from the conversation that's passed to the judge model to help improve the accuracy of a judge for a specific agent’s use case. Evaluator rules give full flexibility regarding when to run these evaluations over live traffic (for instance, running over the full conversation transcript or just individual agent messages within a conversation). Prompt injection detection, custom fulfillment scores, and even identifying conversations covering a certain use case or topic are all configurable within Grafana Cloud.

To supplement LLM-judges, deterministic evaluators like regexes can flag conversations containing a specific word. For example, you could check for conversations recommending a competitor product or leaking an API key. 

Within Agent Observability, evaluators also publish metrics to Grafana Cloud Metrics, enabling custom dashboards to go along with the aggregate dashboards already there. Having the metrics in Cloud Metrics also unlocks alerting through a dedicated Grafana Alerting integration. That way, if a change is pushed to the agent that tanks performance in front of production customers, the on-call engineer can be alerted to revert it.

Once evaluators are monitoring live traffic, developers can have more trust in their agent out in the wild and more confidence that issues will be proactively identified.

The next step is supporting iteration on the agent without introducing regressions.

Phase 3: Building a more robust test suite

As mentioned in Phase 0, you can establish enough confidence for your first deployment through benchmarking and building a test suite of core reference use cases that the agent needs to handle. But this should not be the end of the story.

Test suites are ultimately just collections of test cases. A simple test case consists of an initial conversation prompt and the expected output result. These are used as the seed prompts to test an agent and give a rubric for responses to evaluate for improvement or regressions.

A conversation that fails or gets a low score from an evaluator is especially valuable. These failures can highlight edge cases and broader areas of necessary improvement for an agent, showing developers where to focus their efforts. Evaluators can be assigned an action to identify conversations that fit these patterns and route them into a collection. Once this collection of conversations that fit the pattern has been assembled, it can be annotated by a human or agent and pushed into a test suite.

Once you have a strong test suite foundation of core use cases, failure cases observed in production, and ideas in mind to improve the agent, you can start kicking off offline evaluations and experiments.

Phase 4: Tracking experiments and moving to CI/CD

"Offline evaluation" is a term for evaluating an agent against a set of test cases and producing a report on its performance. This is different from online evaluation, where we are evaluating live traffic as it comes in. Offline evaluation is helpful for initial agent development, benchmarking, validating changes to an agent, and CI/CD.

Benchmarking helps determine if an agent or standalone LLM performs well at a certain, well known task. For example, Grafana’s o11y-bench measures how well an agent performs over a suite of ~70 tasks related to observability within the Grafana ecosystem. As new frontier models are released, benchmarking can help inform which models are best suited for which tasks relevant to your industry.

Verifying agent changes and CI/CD are more relevant to developers actively iterating on a agent. Having a platform that makes it easy for both developers and product managers to make changes to the agent and see how the agent performs over the test suite is crucial for rapid iteration. The most common use case is comparing an agent’s performance over the test suite before and after a prompt change to determine if the change is safe to deploy to production. 

Run Experiments using the agento11y SDK to pull the test suite, then invoke the agent using the starting prompt, record the scores, and assemble a report on the agent's performance. Test cases are identified by a unique ID that can be used to compare performance across two or more runs that include the same test cases. If you take an example with o11y-bench, using Haiku as a baseline, it’s clear that Sonnet is a vast improvement in quality for many of the test cases, but it increases the cost:

 (Try it out by publishing your own o11y-bench run to Grafana).

You can trigger experiments manually or use them as CI/CD gates for pull requests to an agent repo for additional peace of mind. A baseline requirement for a pull request could be running an experiment in Grafana to prove there are no regressions, then pasting the link in the PR description.

Once satisfied with results, the change can be merged and deployed, and the agent version can be monitored in the agents tab with the diff, in case a rollback is needed.

Repeat and Iterate

The final step is to repeat! Repeat the process of gathering data, improving the test suite, and testing out changes to the agent grounded in real data. As mentioned above, the field is constantly evolving. New models are released, new harness techniques are developed, and there is an infinite supply of prompt and tool configurations that could improve your agent. 

Try Agent Observability today

Traditional observability has always been about maintaining and improving the way software runs. Agent observability takes it a step further by enabling developers to iterate on complex, non-deterministic agents by grounding results in real data.

And with Grafana Agent Observability, which is now generally available in Grafana Cloud, you'll get peace of mind knowing you've got the tools you need to monitor and improve your agentic workloads. Check out the docs to learn more.

Grafana Assistant is the easiest way to get started with metrics, logs, traces, dashboards, and more in Grafana Cloud. We have a generous forever-free tier and plans for every use case. Sign up for free now!