Grafana Labs at KubeCon: OpenTelemetry eBPF Instrumentation, Prometheus deep dives, and more

KubeCon + CloudNativeCon North America 2025 is right around the corner — and we can’t wait to see you there! 

The event will run from November 10-13 in Atlanta, Georgia, bringing together open source enthusiasts for the Cloud Native Computing Foundation’s flagship conference. And this year’s event is a particularly special one, marking the CNCF’s 10-year anniversary.

Grafana Labs is a proud silver sponsor of KubeCon North America 2025, including Observability Day 2025.

If you plan to attend, drop by booth #444 (hall B3) at KubeCon’s sponsor solutions showcase to meet the team and grab some swag (because who wouldn’t want a sweet sticker of Grot holding a Georgia peach?!).

You can also contact us to set up a meeting with one of our experts, or attend one of the sessions we’re participating in (more on that below).

KubeCon sessions featuring Grafana Labs

The following KubeCon North America talks (all times local) feature, or are led by, members of the Grafana Labs team:

Telemetry that matches your model, verified live

Monday, November 10 • 10:40am - 11:05am EST 

OpenTelemetry semantic conventions define how telemetry should look, but do instrumentations actually follow them in practice? Your organization might define additional conventions — but how do you ensure instrumentations stay up to date as conventions evolve?

What if you could get a comprehensive report on the actual telemetry your application produces, checked against your conventions model?

In this session, Liudmila Molkova, Staff Developer Advocate at Grafana Labs, and Jeremy Blythe, VP Engineering at Evertz, share OpenTelemetry Weaver’s live check: a validation tool that listens to telemetry in real time and analyzes its compatibility against formally defined conventions. They’ll show you how to run live checks in your CI/CD pipelines to catch instrumentation issues before they hit production — no more guessing whether your telemetry matches your model.

Beyond basic model compliance, the speakers will also demonstrate how to write Rego policies to enforce organization-specific checks and best practices.

Straight into the deep end! Learning Kubernetes and cloud native from scratch in late 2024

Tuesday, November 11 • 5:45pm - 6:15pm EST 

Many people attending KubeCon have spent years on both cloud native and Kubernetes — growing in their technical knowledge alongside the development of these concepts and technologies.

But, for others, this isn’t the case. There are those who are just starting out their careers. And there are others who pivoted from industries that used older methods, tools, and technologies.

This talk, led by Grafana Labs Senior Principal Field Engineer Éamon Ryan and Senior Field Engineer Ayah Elshaikh, aims to take those very people through what it’s like to dive head-first into Kubernetes and cloud native in the modern day through the eyes of an engineer who did exactly that just recently — learning it in its mature form instead of iteratively along with it. It discusses the challenges and pitfalls that can be overcome or avoided.

Additionally, the session covers what’s changed in those years that has made things easier, from the alternate perspective of an engineer who has been using Kubernetes for many years.

UX Research Report: Prometheus and OTel’s resources attributes

Wednesday, November 12 • 2:15pm - 2:45pm EST 

Prometheus and OpenTelemetry are two CNCF projects that focus on observability and truly excel at their main purposes. However, it’s no secret that they started their integration journey on the wrong foot. This story is getting better over time and being done in a data-driven way!

Through CNCF’s LFX mentorship program, the presenters of this talk — Victoria Nduka, User Experience Designer, Independent and Amy Super, Principal Product Designer, Grafana Labs — conducted UX Research to understand how Prometheus should handle OTel’s resource attributes. Using quantitative and qualitative approaches, they collected opinions from co-founders of both projects, active and old maintainers, and several end users.

By joining this talk, you’ll learn more about what is going well and what could be improved, while listening to Victoria and Amy’s educated suggestions for the project maintainers.

Debugging your cluster when it’s on fire

Wednesday, November 12 • 4:00pm - 4:25pm EST 

Imagine you have a Kubernetes cluster that’s hosting some number of services. Perhaps these services are written in different programming languages, and perhaps there are some databases in the cluster, too. Now imagine that this cluster is intermittently experiencing errors and it’s not easy to tell what’s going on.

In this talk, Nikola Grcevski, Principal Software Engineer at Grafana Labs, and Tyler Yahn, Senior Software Engineer at Splunk, will show you how you can add detailed telemetry immediately to a problematic production environment cluster, without any changes to your existing cluster configuration or applications, with the new OpenTelemetry eBPF Instrumentation project. They’ll show you how you’ll be able to get insights into what’s wrong with your cluster or services by leveraging on-demand distributed traces and connectivity graphs, even if it’s the first time you have heard the term OpenTelemetry.

They’ll also discuss the design principles that make this technology safe to deploy in an already problematic environment, without further compromising the stability of your cluster.

Observing dark matter with OpenTelemetry

Wednesday, November 12 • 5:30pm - 6:00pm EST 

Stories of time wasted (and lots of emotions). What time do we really measure when we instrument our applications and try to collect request duration times? What conclusions do we draw when the request duration reported by client requests doesn’t even come close to the times reported by the server? Do we blame our network, our reverse proxies, or our monitoring tools when “dark matter” gaps exist in our traces?

This talk, led by Mario Macías, Staff Software Engineer at Grafana Labs, and Sam Alipio, Staff Product Manager at Pasteur Labs, covers some subtle nuances in request duration measurement, which can often lead to completely misleading results. They’ll dig deep into the OpenTelemetry instrumentation approaches in a few popular programming languages and correlate what the duration timings in traces or logs mean.

They’ll show how using the newly donated OpenTelemetry eBPF Instrumentation for application instrumentation allows you to augment your telemetry to overcome various inaccuracies in duration measurement… and get closer to the truth.

Prometheus intro, deep dive, and open Q+A 

Thursday, November 13 • 1:45pm - 2:15pm EST  

As the second oldest project in the CNCF, Prometheus is probably something you’ve heard of before. Nevertheless, in this talk, Owen Williams — Prometheus project maintainer and Principal Software Engineer Grafana Labs — will give you an introduction from the very beginning, followed by a deep dive into the exciting new features that have been released recently or are in the pipeline. You will learn about many opportunities to use Prometheus, and maybe even be tempted to contribute to the project yourself.