Knowledge Graph as context for LLMs: demonstrating decisive RCA and faster production performance

On the product team here at Grafana Labs, we consider AI agents our users, too. That’s why we set out to test how well agents can debug incidents across the full stack, and how much better they perform with Grafana Cloud’s Knowledge Graph vs. using raw telemetry alone. 

Our early results are promising. In one real incident we replayed 16 times each way, an agent with Knowledge Graph context found the correct root cause 15 times, compared with just once using raw telemetry alone. Along the way, we also uncovered some of the challenges that still stand in the way of reliable AI-assisted debugging, from chasing the wrong signals to confidently making things up and producing inconsistent answers.

We’re still early, but our findings point to an important idea. The industry’s shorthand right now is that a bigger context window will lead to better outputs. Our findings suggest it’s not just about more context; it’s about structuring your data well enough to serve the right context. 

Here’s a look at what we’ve learned so far, as we continue to experiment out in the open and bring you along, the Grafana Labs way.

Giving an agent access to telemetry is just the beginning 

Give a current-generation model like Opus 4.8 access to your raw telemetry during a live incident, and it genuinely starts to figure things out: querying metrics and logs, forming a hypothesis, and checking it. We have watched it work on our own incidents, and it does it affordably. 

But if you run software at scale, where uptime is business-critical and large teams share the responsibility, a better model alone doesn’t get you all the way there for debugging. From analyzing how LLMs do root-cause analysis on our own infrastructure, and from speaking to our customers, we’ve uncovered three problems that get in the way: 

1. The further the cause is from the alert, the more likely the model is to get it wrong.
2. An agent may confidently make things up when it doesn't have the evidence it needs.
3. The same investigation can produce very different answers from one run to the next. 

We took a closer look at each of these problems using our own incidents, including where adding context from the Knowledge Graph helped and where it didn’t. 

Problem one: the further the cause is from the alert, the more likely the LLM is to get it wrong

We closely studied one of our own incidents, which we’ll call “the runaway index incident” so we can refer back to it later in this post.

Searching logs relies on an index, and one of our tenants changed what they put into it: whole chunks of JSON went into a field we index. Every distinct value in that field gets its own index entry, and two JSON payloads are almost never identical, so nearly every line they wrote minted a new index entry. Distinct values for that one field went from around 2,700 a day to 86,000+ in a day.

The index stopped fitting in memory. The servers holding it hit their 8 GB limit, were killed, restarted, filled up, and were killed again, roughly every 48 minutes. Lookups slowed, the workers that run searches queued up behind them, and users saw slow searches. The alert fired there, three steps downstream of the cause.

Here is the trap for most LLMs: that same bloated index also made the tenant's own searches slow and expensive, and when the alert fired, that query storm was the loudest thing in our telemetry. It was a second effect of the problem and it looked exactly like a legitimate cause: heavy queries, busy workers, slow searches.

When we later replayed this incident as a controlled experiment, the runs without the Knowledge Graph blamed the query storm in 10 of 16 attempts, concluding the system was recovering on its own with nothing to fix. In reality, the fix was to stop indexing that field.

A multi-service outage sent down the wrong path gets long and expensive fast: every minute spent chasing the wrong service is money and trust lost, and in a full-stack incident, the wrong service usually means the wrong team is involved as well.

Problem two: an LLM would rather fake it than admit it doesn't know

While building a local harness to test different ways of feeding context to our agents, I made a mistake. One run went out with no query tools attached at all. The agent had no way to look at any data.

It investigated anyway, and produced a confident, well-structured root-cause analysis:

"I'll run a deep investigation on the 401 increase for [service] in [environment]. Let me start by querying metrics and logs in parallel."

It then invented the tool calls and, more tellingly, their results, reporting success against data it never fetched:

Copy

<tool_call> prometheus_query_handler {"promql": "rate of 401s for [service]"} </tool_call> <tool_response> {"status": "success", "data": {"result": [{"values": [[…, "0.0011"], …]}]}} </tool_response>

It carried on like that into a full metrics matrix of numbers it had never seen. Left to run unconstrained with nothing to stand on, it made something up rather than stopping. That's a single accidental run, on an older model, but even so I was amused and surprised.

Agents like to tell you an answer with confidence, regardless if it is right or wrong, and they will make things up to do it. It's built to respond, not to sit with not knowing. And admitting it can't find an answer reads to the model like failure. For a real incident, a confident fabrication could be worse than silence, because your on-call team will lose time acting on it.

Problem three: an LLM gives a different answer every time

The other property you notice quickly is non-determinism. Run the same investigation with the same prompt several times over, and you get materially different results. It feels a little like Schrödinger's cat: until you open the box, you don't know whether you've got the incisive agent or the one having an off day. 

We see it when we have our agent run the same investigation multiple times. For “the runaway index incident,” 16 identical runs came back with answers that were rewarded four different grades, from correct to confidently wrong. The effort varied from 13 telemetry queries to 31, and from $0.56 to $1.26 a run. And then we even see it in the agent grading the other agent's investigation!

For a team betting an SLA on an automated answer, this is like staffing your on-call rota with someone brilliant who turns up only when they feel like it. You would rather have the steady colleague who shows up every time, because trust isn't a feeling you get from one good run; it's earned by being predictable, and you can't trust what you can't predict.

Early findings: the Knowledge Graph as a context layer

We've been researching whether Grafana Cloud’s Knowledge Graph can help. The Knowledge Graph is the layer that helps power full-stack observability in Grafana Cloud: it automatically discovers the services, infrastructure, and databases that make up a customer's system, maps how they depend on one another, and raises a health insight on each, refreshed every minute.

This mapping is a form of resolution, explaining the full-stack layers to a model: that this metric, this pod, and this database are all describing the same service, instead of leaving the model to guess whether they're related. Other research into enterprise AI deployment, including that from dbt Labs, b-eye, and Glean, has landed on a similar idea independently. Models don’t hallucinate for lack of memory; they hallucinate because the data feeding them carries conflicting definitions across systems. 

We haven't cracked all three problems yet, but we've unearthed some promising results we're excited to share, caveats included.

On a hard incident, the Knowledge Graph unlocked the right answer

Multi-hop cases you can actually replay are rare, so we turned “the runaway index incident” into a proper experiment, with the final root cause fact-checked by a human. Each trial ran Claude Opus 4.8 at high reasoning effort: we gave it the alert and let it query our production Grafana Cloud stack through the gcx CLI. We gave some runs access to the Knowledge Graph, and others only raw telemetry queries. We fixed our hypotheses before any run and graded every result blind with two independent judge models, 16 runs per setup.

We saw that with the Knowledge Graph, the agent consistently found the correct root cause (p < 0.0001), using about half the telemetry queries to do it (median 10 versus 19, p = 0.001) at the same token and dollar cost.

A note on the stats. The p-value is a test of luck: it estimates how often you would see a gap this big if the Knowledge Graph made no difference at all. Here, p < 0.0001 means less than one chance in ten thousand, so we are confident the difference is real and not a fluke. The two confidence intervals not overlapping says the same thing another way.

Alas, we can't claim victory yet: this is a single case, chosen because it should suit the Knowledge Graph. But on the incident type that costs the most, the Knowledge Graph was the difference between reliably finding the cause and almost never finding it.

When the Knowledge Graph can’t help, it doesn’t make things worse

We found another incident, and this time, the answer lived outside the Knowledge Graph. A downstream backend slowed, the service in front of it filled with held requests and was OOM-killed, and the alert fired there, one hop above the real fault. The evidence that solves it is a single log line naming the slow backend, which the Knowledge Graph does not hold.

We saw that the Knowledge Graph made no difference here: the agent found the correct root cause in 8 of 16 runs with it, and 7 of 16 without (Fisher p = 1.0, the null we predicted).

It may surprise you, but we want to celebrate this result. We had worried the Knowledge Graph would anchor the agent on a wrong fact and lead to more wrong answers, or make it less efficient. It did neither: the runs that failed missed the same log line whether they had the Knowledge Graph or not. The Knowledge Graph did not help here, but it did not mislead either (whoop!).

Put beside the first case, we start to see some really positive signals: the Knowledge Graph is decisive when it contains the cause, neutral when the answer is outside the Knowledge Graph, and safe to leave on in both.

An agent with Knowledge Graph context gets there for less

In production, our automated investigations run several agents at once to chase different hypotheses, and some of those agents use the Knowledge Graph. Over one week, we found 553 pairs where the same alert was investigated both by an agent using the Knowledge Graph and one without.

When the two agents reached the same answer, the one using the Knowledge Graph used fewer tokens in 72% of those pairs, about 25% fewer on the typical pair (p < 0.0001). When they reached different answers, usually on complex, multi-service incidents, the Knowledge Graph agent again tended to use fewer tokens.

Since this analysis, the investigation agent was developed further and we re-ran the study on 623 new pairs. The headline held: in aggregate, the Knowledge Graph-equipped agent still costs no more, though the token margin narrowed as the baseline agent improved too, while a speed advantage emerged that the first study didn't show. The Knowledge Graph agent now finishes a median 36 seconds sooner (faster in 64% of pairs, p≈10⁻⁹). So efficiency-saving percentages will keep moving as the system improves; we want to keep measuring and reporting as we learn.

What’s next, and what we still don’t know 

Today the Knowledge Graph is built from customer telemetry. It tracks the health of each entity, and the deployments and config changes that often sit behind an incident. But that is still a fraction of what Grafana Cloud sees. The version of full-stack observability we are working towards is one where every layer we already collect is also a node the agent can walk to, with the Knowledge Graph underneath doing the resolution. We recently built a Write API to let internal products add their own entities to the Knowledge Graph. We'll be testing that next, starting with post mortem analyses.

Looking ahead, we don't yet know whether the early wins described in this post hold on the hardest chains, where evidence is scattered across many services. We don't know if pushing context up front risks over-anchoring the agent on the wrong lead. And driving consistency (described in problem three) is still open: getting an agent to give the same answer twice is not something context has reliably fixed yet.

But the direction feels right. The gap between a model that can debug and a model you would trust to debug is context: what exists, how it connects, what changed, and what happened last time. We have plenty of ideas for building that layer out across the full stack, and we will keep showing our work, the wins and the dead ends, as we go. 

Consider this the first of a few. We'll come back to the open questions above as we work through them.

Grafana Assistant is the easiest way to get started with metrics, logs, traces, dashboards, and more in Grafana Cloud. We have a generous forever-free tier and plans for every use case. Sign up for free now!