Grafana Labs at KubeCon: OpenTelemetry, Prometheus 3.0, eBPF, and more

We’re headed back to KubeCon, and we can’t wait to see you there!

KubeCon + CloudNativeCon North American 2024 will run from Tuesday, Nov. 12 to Friday, Nov. 15, in Salt Lake City, Utah, bringing together open source enthusiasts for the Cloud Native Computing Foundation’s flagship conference. Grafana Labs is a proud silver sponsor of this year’s KubeCon North America event, including Observability Day 2024.

If you plan to attend, drop by booth R7 at KubeCon’s sponsor solutions showcase to meet the team and grab some swag (and enter for a chance to win some some special swag if you take our Observability Survey while you’re there). You can also contact us to set up a meeting with one of our experts, or attend one of the sessions we’re participating in (continue reading to find out more).

KubeCon sessions featuring Grafana Labs

The following talks (all times local) feature, or are led by, members of the Grafana Labs team:

But wait! There’s…still more? - Observability data volumes and strategies for managing them

Tuesday, Nov. 12, 10:40 - 11:05 a.m.

As we add more observability signals and points of insight into the software we design, we create more and more data. Data that has to be processed. Data that has to be stored. Data that has to be iterated over and retrieved. All of this increases stress on the system that stores it, which naturally increases the cost to run the system, in terms of infrastructure, people power, and more.

This talk, led by Éamon Ryan, Grafana Labs Senior Principal Field Engineer, takes people through the history of how these volumes have grown over time, then moves into the current strategies and tradeoffs for managing them. It also offers a glimpse into what might come into this space in the future.

So you want to write memory with eBPF?

Tuesday, Nov. 12, 4:30 - 4:55 p.m.

A lot of eBPF programs fall into the category of observing the Linux system, i.e., the kernel, system libraries, or user-space programs. For the purpose of observing the system, we mostly rely on reading memory with eBPF, either kernel or user-space memory. However, sometimes various eBPF use cases require writing memory—for example, propagating W3C context for various application protocols.

This talk, by Nikola Grcevski, Grafana Labs Principal Software Engineer, and Mike Dame, Odigos Senior Software Engineer, focuses on the journey to implement W3C trace context propagation with eBPF at various levels of the protocol stack. They explore what memory write eBPF APIs are available to us today, along with their implications on system security, stability, required permissions and implementation difficulty. They’ll present two working solutions with their pros and cons, a lot of dead ends, as well as explore what a new approach might look like by leveraging the “BPF arena” feature in kernel 6.9.

Unlocking cost savings and new possibilities: your guide to Prometheus Remote Write 2.0

Wednesday, Nov. 13, 11:15 - 11:50 a.m.

Prometheus Remote Write is the protocol used to send Prometheus metrics from Prometheus or any other metric source to compatible remote storage endpoints. Remote Write is generally used for metric long-term storage, centralization, and cloud services. It also enables users to run Prometheus in an agent mode, reducing local storage requirements.

Welcome to Remote Write 2.0! In this talk, Bartłomiej Płotka, Google Senior Software Engineer, and Callum Styan, Grafana Labs Senior Software Engineer, introduce you to the next iteration of the popular protocol, which adds more functionality while cutting your egress costs up to 60%, and keeps the previous version’s easy-to-implement stateless design! The audience will learn what’s changed in the second version of Remote Write, what it unlocks, and how easy it is to update or adopt. Finally, the speakers, who are Prometheus maintainers and RW2.0 spec. co-authors, will share the latest benchmarks and differences with the common alternatives.

OpenTelemetry project update

Wednesday, Nov. 13, 3:25 - 4 p.m.

This is the official OpenTelemetry session at KubeCon, and it’s led by:

• Morgan Mclean, ​​Director of Product Management, Splunk
• Juraci Paixão Kröhling, Software Engineer, Grafana Labs
• Daniel Dyla, Senior Open Source Architect / OpenTelemetry GC, JS, Maintainer, Dynatrace
• Ted Young, Director of Developer Education, ServiceNow
• Alolita Sharma, Cloud Observability & Infra Engineering, Apple

OpenTelemetry started with distributed traces and metrics, but the project’s vision has always been to provide whatever signals are needed from infrastructure, services, and more. This session will focus on what’s coming next, including new signals and sources. Join to learn about OpenTelemetry’s new logging functionality, including its two logging paths, the benefits of each, and real-world production examples. They’ll show the power of the next wave of OpenTelemetry enhancements, including profiling and the insights that this unlocks in combination with distributed traces, and how we’re extending your observability to client applications. They’ll wrap up with a Q&A of 10+ project maintainers who can speak to these topics and more.

Watching the watchers: How we do continuous reliability at Grafana Labs

Wednesday, Nov. 13, 4:30 - 5:05 p.m.

Nothing is foolproof. Everything fails eventually. Observability tools help predict and lessen the impact of those failures, as the watchers of your software systems. But who watches the watchers? At Grafana Labs, we’re not immune to production incidents. Just like any company, we still sometimes move too quickly. We run complex, microservices-based systems ourselves, so we have to eat our own dog food on a daily basis.

In this talk, Nicole van der Hoeven, Grafana Labs Senior Developer Advocate, reveals:

• How we solved a years-long mystery that cost us $100,000+
• How we got our internal Grafana Mimir clusters to reliably hold 1.3 billion time series for metrics
• What we’ve had to do to scale our Grafana Loki clusters to handle 324 TB of logs a day
• What our Grafana dashboards to monitor Grafana Cloud look like

Sometimes, it’s easier to learn from failures in observability than from successes. This talk is a confession of some of our worst sins as well as a realistic look under the hood at how we’re improving the continuous reliability of our stack.

Celebrating Prometheus 3.0: A deep dive with the maintainers

Thursday, Nov. 14, 2:30 - 3:05 p.m.

Prometheus is an open source monitoring system and a CNCF Graduate project. It benefits from a rich ecosystem, including Alertmanager, efficient client libraries for many languages, the Prometheus Operator to install on Kubernetes, and numerous exporters to provide the raw data. This year, Prometheus releases the 3.0 version, which includes new features, a refreshed UI/UX, and plenty of new things that build on what has worked well for years.

Join Josue (Josh) Abreu, Grafana Labs Principal Software Engineer and Richard “RichiH” Hartmann, Director of Community at Grafana Labs, to celebrate the 3.0 version and learn what it enables for new and existing users, how to upgrade, and how to get the most out of the latest version! Prepare tons of questions; they’ll have a lot of interactive time for questions and they want to hear community feedback!