Agentic observability: Lessons on speed, judgment, and trust from Grafana Cloud customers

Company: thredd
Industry: Financial Services
The dashboard used to be the first stop during an incident. At thredd, an AI-first issuer processor, that's changed.
In this fireside chat, Teodor Baltaretu, Site Reliability Engineer, and Mark Swarbrick, Senior Platform Engineering Manager, join Robin Gustafsson, VP of Product at Grafana Labs, to share what that change actually require, including months of unglamorous work standardizing naming conventions, tagging every metric, trace, and log by team and service, and building the data foundation that makes AI useful rather than dangerous.
They share the first success that turned skeptical developers into evangelists, the cost-reduction recommendation that taught them context is everything, and where they've drawn the line between what AI can act on and what stays with humans. In payments, that line matters. "Everything that could cause problems on the customer side should be handled by humans," Baltaretu says. Everything else is fast becoming part of the daily routine.
Speakers

Teodor Baltaretu
Site Reliability Engineer, thredd
%3Aquality(90)%2F&w=3840&q=75)
Mark Swarbrick
Senior Platform Engineering Manager, thredd

Robin Gustafsson
VP of Product, Grafana Labs
Robin Gustafsson: All right. So we're on the final stretch as Carl mentioned. So I have the pleasure of speaking with Mark and Teo here today. And as we've been getting to know each other a bit since we met for the first time yesterday, we've had some very interesting conversations out in the exhibition hall and over here just now. And I've been telling them I always look forward to these sessions because I think there's so much learning to be had listening to how our customers are actually applying the products that we like to talk about, but how they're actually being applied in real world use cases. And that's many times where I think people in the audience also sort of can see themselves and feel a familiarity in the both successes and struggles. So thank you Mark and Teo for joining me today. And we're going to start off by having you introduce yourselves so we know who you are.
Teodor Baltaretu: Yeah, I can start. Hello everyone. I'm Teo. I'm an SRE at Thread. And basically my role is to make sure that all our products are reliable, observable and ready for production. I work a lot with all different engineering teams to make sure that, I like to say it, we improve the developer experience. So basically everything we do, we are working in the payment industry. So of course reliable is the most important thing for both us, but also our clients.
Mark Swarbrick: Cool. Thank you. Yeah, my name's Mark Swarbrick. I'm the senior platform engineering manager here at Thread. My responsibility is looking after platform engineering team as you can imagine, but I also am trying to build out the SRE function and also look after application support. So the company I work for, Thread, we're AI first issuer processor. So I always try and explain it to everyone, my parents, for our customers, if you tap your card, we process the auth and clearing transactions for our customers. So I'm not even kidding where we have some customers where we literally drop two or three auths and we have them on the phone. So the focus on quality observability and uptime is really paramount for what I've come into. My background, I've worked for FinTech startups. I've worked in the open source space with MySQL and I was really spoiled at my last place where it was a greenfield fintech startup and it was like everything you could want, time and money and no live customers.
So I was really spoiled to work some really good DevOps engineers where I was really introduced to what Grafana could do. Came into Thread and we've had the typical problems of migration from bare metal and all of the issues you get with legacy systems and migrating to cloud native. So the journey we're on now is really just to bring ourselves up to speed and leverage what we can do with Grafana. So that's me.
Robin Gustafsson: Yeah. Great. Amazing. And so we're going to learn a little bit about what your journey has looked like as you've sort of overcome some of those challenges. I'll just tell everyone as well. So my name is Robin. I lead the product organization at Grafana Labs. You know who I am as well. And if you have any questions throughout the sessions, you have a QR code as you had for the other sessions as well. So please add any questions and we'll get to them towards the end. But before we get into the AI and Agentic side of things, maybe you can start telling us how you got started with Grafana Cloud and where you're at today with Grafana Cloud.
Mark Swarbrick: Absolutely. I mean, for me, I came in and looked at Platform and looked at what we had. And again, we were rebuilding some of our legacy services on a new AWS cloud native platform. But really what we ended up with, with a lot of siloed teams that have all implemented observability in a slightly different way based on what they've done. So we had a mix of AWS Grafana, Xray, CloudWatch and various different technologies in a different way. We had legacy CoreLogics for our legacy stack. So we really had a massive mix of different technologies implemented in a different way. And really it was disjointed. We didn't have, I say didn't, we still don't have all the traceability I want all the way through and all the right metrics and observability and very log focused. So there was definitely a journey that we needed to be on and it's still are on to get to that point.
But for me, one of the key things we needed to address quickly was to pick an observability platform and try and unify the whole company in different areas on that platform. So this is where Teo and the rest of the team came in trying to pick that platform. And I feel based on my experience previously, I was really happy that everyone together said, yeah, Grafana is the obvious choice native hotel and the feedback from developers has been really good as well.
Robin Gustafsson: And Teo, from your perspective as an SRE, what things from a product perspective do you think Grafana Cloud has helped you and your team with?
Teodor Baltaretu: Well, we are coming from a lot of legacy places and a lot of observability tools we had at Red before. So our goal was to have one tool for observability. And where we are looking at different tools, we started having a pilot trial with Grafana Cloud where our idea was we take one engineering team, we put it into Grafana Cloud and we see exactly first if those features that us as the SRE team we think are useful. We can say that any tool is great, but more important is also what the engineering teams think about it. Because me from a technical point of view, I can say, oh, those are some great features. We have those beautiful things that we're using both on a cost control side or just for example, panels and dashboards, but is most important on the engineering side. So our idea was pilot and first one engineering team, test Grafana Cloud for everything.
And the success was great that we continued with not just one team.
Robin Gustafsson: Yeah. So the initial success led to credibility that you could continue the rollouts to more of the organization.
Mark Swarbrick: Yeah. And what was even better is we had a fixed budget for that. And every day tier is like, I've found this cost reduction, we can use adaptive logs for this and this. And we ended up actually getting more and more products adopted onto Grafana under the same budget before I had to go into the begging bowl to actually change our vendor and get more funding through. So it was great being able to get more and more people on board.
Robin Gustafsson: Yeah, that's great. So I think everyone has started to use more and more LLMs and agents in our work. So what does that look like at Thread? Have you seen anything in terms of stability approach that's changed? What's been the differences?
Mark Swarbrick: Yeah, I'll start at the beginning in terms of Thread and then go into more specifics. So I think like any company, there was the, "Oh, AI quick.
Robin Gustafsson: We
Mark Swarbrick: Need to adopt, we need to use it." So I think there was a scrabble around between SecOps throwing their hands up in the air and us wanting to drive adoption, but also trying to be AI first and using it in the right way. So luckily I think we managed to adopt quite quickly and get a proper SecOps policy around the adoption of AI. And we chose to use Claude for that. But at the same time there was, as we're building observability out, it was like, ah, okay, we can start leveraging the Grafana side as well, which
Teodor Baltaretu: Is
Mark Swarbrick: What Teo's been looking into.
Teodor Baltaretu: Yeah. I think we are just lucky with timing because we wanted to migrate from so many different tools to one tool at the same time with the rise of AI. And we had differences in terms of how observability was before and how observability is now and hopefully how observability will be in the future. Because in the past we will have an incident happen and people would either look at a dashboard or look at the logs directly. But right now with how we are doing observability inside Grafana Cloud, now we have metrics, traces, logs all together. And we also have AI who is built into it, has its own skills and has the knowledge based on our observability. And this has led us to new ways of doing incidents and also preparing us for future. For example, when we have any deployment from, we have a lot of environments before production, but until now we didn't have a way to really see changes after a deployment in terms of observability, even cost or everything.
And now we can just point the AI and say, look, we deployed 15 minutes ago to staging, what's happening in staging for service X? And now we see changes and this is used by developers every day and it's helping us drive further.
Robin Gustafsson: Yeah. So you bring up an interesting point there, that making sure the agent has the right context so it can actually be useful in different kinds of situations. Could you maybe help us paint a picture of what things did you, if anything, did you have to change to make sure that context was available and in the right shape?
Teodor Baltaretu: Yeah, of course. This was a long fight we had because we had all services divided into different places. We had Kubernetes, we have on the Fargate side and we had each team have more power before on their observability and this resulted in different standards or naming conventions, anything like that. So the first thing that we have worked on after understanding the power of AI was tagging every metric traces and logs with different things. So for example, important milestone for us was having everything tagged by team or by service so that in retrospective, not just that the AI knows that which metrics are connected to which traces to which logs and it can have all the metadata needed to also create the knowledge graph and have other features that work even better with the right context. But it's also good for a management SRE cost control part because we have worked a lot to have detailed cost tagging based on, for example, a service or a team.
And we have AI look over those cost changes. So for example, if a team deploys something in an environment and the cost goes insanely up, we get pinged and we know that before it reaches production or before it breaks anything.
Mark Swarbrick: And I think for me, it's the boring bit. It's the cost control and all the preamble. I mean, you look at all the amazing features that you can do with Grafana, and this is one of the things I needed to stress to our development teams and the management is there's a lot of toil. There's a lot of just getting the naming conventions right. There's a lot of standardizations and I'd hear TO fizzing. It's like they've prefaced it with the wrong thing and I've told them. And it's that kind of frustration and the unsexy work to get to this point. And I think it's literally only in the last couple of weeks we've got to the point where everything's there and I can even go back to the business and go, each individual development team can now be responsible for their observability costs, which is such a massive thing that we haven't had before.
Robin Gustafsson: Yeah. So a lot of work, but you can now reap the benefits of that.
Mark Swarbrick: Yeah, absolutely.
Robin Gustafsson: So if we talk about the first success that you saw as you can now sort of start using more AI in your observability approach, could you maybe share one of the first successes that you had and what that looked like?
Teodor Baltaretu: Yeah, of course. So the first success was actually, even with the pilot team, we are bringing it into Grafana Cloud because I see it both ways. There was also great success with the use of the AI and the use of investigations. We had problems with the auto collector dropping. We are using a site car in ECS and someone modified a configuration and missed one character and that broke everything. The collector wouldn't be turned on and we had nothing in Grafana Cloud for the specific thing. And the team has investigated through using AI into Grafana Cloud and it found the root cause quicker than the developers would have got it. They didn't think of even the possibility of the collector not working. And right now with MCP, things are getting even better because now we can connect them to the code source directly and you can see directly from PRs and even commits if something has changed.
But I also want to talk about another success, which for me it's even better. It was the same team when they got back at me with feedbacks that were so positive. I thought they were salesmen from Grafana. I couldn't believe how positive they were and how much they thanked me because their experience day to day was so much better in both debugging problems and pushing to production. So for me, that was the point where I went to Mark and said, "This is the way we should go and we should go all in."
Mark Swarbrick: And I think the fact that they were self-serving as well, it's always difficult to get budget to increase your team. And at the moment, SRE is a team of two and a half people. So the ability that you've got developers now that are self-serving their own observability in a standard way is fantastic because that moves more towards the build it, you run it model, which we're trying to get to.
Robin Gustafsson: Yeah, 100%. And also it's always super beneficial to have people experience some initial success. That usually means they have gotten an appetite to continue and get more of those successes.
Mark Swarbrick: And the work gets around, the teams talk to each other.
Robin Gustafsson: And you can start to build more of also a community around sort of observability as a practice in this case. All right. So we talked about successes. Let's talk about the other side. Do you have any interesting sort of failures that you can share that you've also experienced?
Teodor Baltaretu: Yeah, of course. It will be nice to be in an utopian way and say we are perfect and everything. We had a lot of issues, especially at the start with the way not everything was standardized. And we had even funny things like the same environment we called in five ways and everything was wrong in the telemetry part. But I have a spicy story. I think this will be very interesting because it's also with the use of AI and with how we understood telemetry. So we are using adaptive telemetry a lot. It's one of the features that I love a lot. And this was at the beginning of our journey in both AI and Grafana Cloud. And one of the things that we are seeing in Adaptive Telemetry is also a report on the metrics that have high cardinality. And we could see that for this team, they had a high cardinality and my job was also on the cost control pulse.
So we are seeing that this team had triple the cost of other teams. And I just launched an investigation and the assistant recommended me maybe dropping a label called worker ID, which was used by Gunnicorn in their team to just assign IDs to their workers. They were using like four workers at the same time. Anyway, the AI had the right ideas why we should drop it, high cardinality. It brings nothing. We don't have it in dashboards, alarms, anything. And of course I dropped it and everything was fine. A couple of days later, the team pushes into production. There were no issues on our environments before. And a couple of hours passed, I get a message that, "Hey Teo, everything is broken." And when I hear that, of course, my heart rate goes to 200. And when we are checking in Grafana and we see that every metric was not broken in the terms that they were not working, but the values were so high, it made no sense.
And we launched another investigation and we discovered that removing that worker ID made so that the four workers, when they were sending the metrics, they were sending to the same metric stream and basically they were overwriting each other. And that led to some numbers going into wrong places. And the solution we got quickly was just to assign specific numbers so that they don't get into out of boundaries numbers and that goes high. But the whole idea was that AI gave us a good solution on the cost control side, but we didn't know about the technical cutoff that we'll have. But now also with the use of MCP and having now all the context, this is why context is important because without context, we couldn't understand why some decisions are bad or not needed.
Robin Gustafsson: Yeah. You can come to the wrong conclusions.
Teodor Baltaretu: Yeah, exactly.
Robin Gustafsson: All right. So I want to make sure that we get to this very important bit, which is the drawing. So like the line between how far you go with using AI and agentic approaches and where you say this is something that humans need to still be responsible for or do. So have you found that kind of line, that kind of trade off?
Mark Swarbrick: I think at the moment we're still very much getting, we want to get closer to that line. At the moment, in my view, I think we still got far too many people who have to stare at dashboards or look at alerts. So for me, I know it's such a cliched phrase, but I want to almost get to the art of the possible and try and do it in a structured way. So at the moment there's a lot of talk about, oh, we could use Claude for this, but it's almost like what AI you connect to what and where's the context and how far we can go? So for me, I want to get to the point where we're not replacing people's jobs with AI, what we're doing is giving them some more context. So if there is an issue, I want as much context related to that alert.
I want them to be prepared and almost even get to the point of preparing a PR. And I think for me, the line is to give people as much information as they can to do root cause analysis and fixes as quickly as they can. I'm still at the point where just to let it press releases and do everything automatically. I'm old enough and gray enough to still not trust the machine I remember terminated to. I'm still a little bit nervous of how far we take it, but I think, I don't know, Tia, I think there's still a lot we can do, right?
Teodor Baltaretu: Yeah. Especially in our domain where we are in payments, I think risk and security are very important. We are trying to make sure that we treat this very carefully. The way I see it and I think how our developers are using it every day, they are using AI to improve things from dashboards to code reviews to everything that can be done without touching production. I think everything that is related to deployment and everything that could cause problems on the customer side should be handled by humans. I think for the observability and the way we're doing stuff, dashboards are good and are amazing when you have the skill to know where to look. And AI is when you don't know. So basically we are trying to incorporate AI at a very careful pace, but it's becoming part of our daily life and routine much more and more every day.
Robin Gustafsson: Yep. Excellent. I see we're rapidly running out of time. I want to make sure we get to some questions. We have a few questions here and if there's any more, please submit them, but let's start with the first one here. So we have a question here. Do you use agentic observability during testing?
Mark Swarbrick: You want to tell them?
Teodor Baltaretu: We do for some teams, but it's still something we are testing right now. We want to explore this part and make it a standard for each team before they go into production, but I think we are still a bit early on that road.
Robin Gustafsson: Yep. Another one here. How would you advise alleviating management's concerns about ballooning costs when trying to get them to go all in on Grafana?
Teodor Baltaretu: Oh, that's a good question.
Mark Swarbrick: Lovely
Teodor Baltaretu: Question.
Mark Swarbrick: I would almost argue that for us, Grafana helps us control the costs a lot more. The fact that we could build cost dashboards, allocate costs to teams. And again, it's like anything, token usage is always an interesting one. We've had the same thing where we've got a token budget and the people that have adopted it have blown through their token budget in a couple of days or something like that. So I think it's managing those cost controls, but also then relating back what the benefits are and what you've achieved and then allowing your budget to grow as the benefits are coming out. So we were hitting our cost cap in CoreLogics, for example, we were hitting a cost cap at two thirds of the way through the month. So I've argued that what we're offsetting with adaptive telemetry, adaptive metrics, and actually having control of those costs, we can then spend on AI tokens, whether it's in Grafana or Claude or whatever we're using to then add value.
So it's just about demonstrating the value. And I found management has actually been incredibly supportive of as long as we can demonstrate the value and we're not just blowing a load of tokens on porting some game across to a different platform or something, we're actually doing something useful.
Robin Gustafsson: You can show that there's some control on the cost, et cetera, and then you can show the value.
Mark Swarbrick: Yeah. And I think already they're saying that.
Robin Gustafsson: Yeah. All right. Can you describe the path process to standardization that you described?
Mark Swarbrick: Pain. Pain and toil. A lot of grafts to be honest, but I think a lot of it was once we'd won the mind share of that individual development team, I think Tio, people were actually coming to you, weren't they, and saying, "What do we do? How do we adopt this?"
Teodor Baltaretu: Right? Yeah. It was also thinking what we should do on both the engineering side and also on the platform side. We started by using the hotel standard for naming and stuff. Then we were looking at different teams and noticing bad things like Mark mentioned with adding specific prac fix to some metrics. And then we -
Mark Swarbrick: You got very upset, didn't you?
Teodor Baltaretu: Yeah, too upset about that. And then tagging everything so that it has the specific metadata that we want so that we have metrics traces and logs in everything because before we had teams with just metrics or just logs or just not everything that we should do.
Mark Swarbrick: So really just a lot of toil to get to the point where we can leverage the features, I think is the realistic view. There's the unsexy stuff that we've had to wade through to be able to now just get to the point where we've got observabilities into a point. We've got the raw data there in a standardized way that we can start leveraging all the cool stuff.
Robin Gustafsson: Yeah. Amazing. Thank you so much, Mark and Teo. Round of applause for Mark and Teo up here.
Mark Swarbrick: Thank you.