Have you ever wondered how to introduce that quick fix you always wanted to do to Prometheus or Alertamanger server? What about proposing bigger changes to Prometheus like improving TSDB storage, API or Prometheus standards likePromQL, OpenMetrics or Remote Write?

It might be easier than you thought! In this workshop, we will, together, propose an example (or yours!) code change to Prometheus! The participants will learn and exercise:

• How to find various code components in the Go codebases for Prometheus and auxiliary projects like Alertmanager, avalanche, etc.
• Proposing the bigger changes through the Prometheus proposal process.
• Testing and benchmarking Prometheus.
• Documenting changes.
• What it takes to become a maintainer one day!

Prerequisites for the active participation: Linux or Mac dev machine, git and Go 1.23 installed.

Born at SoundCloud and now a graduated CNCF project, Prometheus has become the de facto standard for monitoring and alerting in Kubernetes and beyond. It benefits from a rich ecosystem, including Alertmanager, efficient client libraries for many languages, the Prometheus Operator which can be installed on Kubernetes, numerous Exporters to provide the raw data, and even projects to run it at a massive scale.

Last year Prometheus released version 3.0, which includes several new features and enhancements, and a refreshed UI/UX. Join two Prometheus team members, Fiona and Saswata, to learn what it enables for the community, the ongoing upstream progress on v3 features, and how to get the most out of them. Learn how to get involved with new v3 initiatives, share feedback, and get an opportunity to have your questions answered!

With the recent stabilization of the OpenTelemetry semantic conventions for databases, it’s an excellent time for OSS libraries to provide users with the observability they’ve been seeking. This talk dives into how you can instrument your application with OpenTelemetry SDKs to improve observability and collect actionable telemetry data from your databases. Learn about the SDK implementations that are currently available by language and database, their current gaps and how you can contribute and develop missing instrumentation.

Whether you’re an SRE, developer, or database administrator, this talk will equip you with the tools and knowledge to bring clarity and efficiency to your database systems.

A robot may not harm humans. A robot must obey humans. A robot must protect its own existence. These are Isaac Asimov’s three Laws of Robotics, created to govern the ethical programming of artificial intelligences. From the Butlerian Jihad to Skynet to cylons, we’ve been immortalizing our collective nightmares about artificial intelligence for years. But there’s an unmentioned law that comes as a prerequisite to all of that: a robot must be observable.

In this talk, I discuss the different types of AI, the factors that make observing AI different from observing applications, and the telemetry signals specific to AI that we might want to listen to. How do we deal with large data sets? How do we observe for model drift? How do we take into account the costs of LLMs? How can we use distributed tracing to follow event sequences? Part cautionary tale and part technical demo, this talk shows how to instrument and monitor AI apps using OpenTelemetry, Prometheus, OpenLit, and more.

Imagine you have small 4-core nodes and larger 64-core nodes in the same cluster, and a DaemonSet that does much more work on the larger nodes. How do you set resource requests and limits appropriately?

Managing resources for workloads deployed as a DaemonSet in Kubernetes can be challenging when load is not evenly distributed across nodes. Static allocation can cause over/under-utilization and scheduling issues. VPA helps, but currently assumes uniform load across all pods, which is a bad assumption for certain types of workloads.

We will discuss our case studies, why this feature will be useful, how our prototype implements per-pod VPA for DaemonSets to improve resource efficiency, stability, and eliminate the need for manual tuning. This is your chance to learn about this upcoming feature and connect with the people who are implementing it!

Have you ever wanted to play an actual game on your observability stack? Well, you can. Not only does Doom run on Grafana, we also built an actual text-based adventure game.

Join us to play a real text-based Observability adventure game! Armed with the tools of the trade—metrics, logs, and traces—you’ll learn to navigate the labyrinth of debugging and optimization, rescuing your application from the clutches of the dark wizard!

In this interactive session, we will dive into a game played live to showcase how each telemetry type is used to solve real-world Observability challenges. As players encounter obstacles, they’ll wield the power of OpenTelemetry to gather critical data and use OSS tools like Grafana, Loki, Tempo, and Prometheus to make informed decisions.

A deep dive into the journey of pushing Prometheus beyond its performance limits. This talk offers an insider’s perspective on scaling a single Prometheus instance using a powerhouse 128-core machine with 4TB of RAM, and processing a staggering 500 million metrics at its peak. It’s a story packed with lessons, insights, and actionable takeaways from operating one of the industry’s largest Prometheus servers.

The talk will go through:

• Breaking Boundaries: Explore the challenges encountered in Prometheus’ design and how they navigated them.
• Diagnosing Bottlenecks: Discover how to combine observability signals—metrics, profiles, and traces—to pinpoint and overcome performance roadblocks.
• Building Resilience: Uncover strategies to optimize metrics volume and enhance Prometheus’ reliability under load.

This session isn’t just about pushing technology to the edge—it’s about learning to work smarter, build better systems, and create a more resilient observability stack.

Prometheus and OpenTelemetry are two CNCF projects focusing on observability and truly excelling at their main purposes. However, they take slightly different approaches, and making both projects work well together has been challenging.

In this talk, Arthur and Juraj, both active contributors to Prometheus and OpenTelemetry communities, will present all the usual frustrations that a user would face when integrating Prometheus and OTel, and all the work done by the OpenTelemetry-Prometheus SIG (Special Interest Group) in the past year to transform Prometheus+OTel into a love story.

You’ll leave this session understanding the core philosophical differences between the two projects that make interoperability so difficult, the progress made to improve the situation, and what to expect in the near future.

The Kubernetes API is awesome and so tempting to use, especially when building Observability Solutions. Nobody wants to just get raw IP addresses and ports in their network or request telemetry, it’s much better to see your pod and service metadata. But what’s even better is that getting information about all the nodes in your cluster can help you produce amazing service graphs.

This talk is a story of how we took down the Kubernetes API in our biggest production cluster at Grafana, by deploying observability tools which make heavy use of the Kubernetes API. We’ll show you the techniques we used to avoid repeating our mistakes, by applying configuration changes and building services which helped us shield the Kubernetes API from the information thirsty observability tools, while keeping the functionality intact.

Traditionally, monitoring your network connections required from devices being able to export the flows data. With the rise of software-defined networks, the responsibility of providing observability capabilities relied on the SDN providers or on software-based packet analyzers that often have a noticeable impact on the cluster’s performance.

eBPF is presented as an efficient, non-invasive mechanism to observe different layers of clusters’ network, from L3 to L7, and automatically extract relevant information without having to redeploy neither the network infrastructure nor applications.

Our talk explains the Grafana journey to provide plug and play network and services observability: how we connect to different layers of your services infrastructure to provide from how network packets flow through your system to the details of the L7 service/client requests and responses, and how the low-level network information is matched with Kubernetes metadata for improved user data navigation.

Designing access control that works seamlessly for individuals and scales to millions of resources is a complex challenge.

From lackluster search performance to feature inconsistency and multi-tenant schema discrepancies, there’s no shortage of issues to face.

Join the Grafana Access squad’s journey through the ups and downs of how we’re tackling these issues using OpenFGA, a CNCF sandbox project, by porting our existing access control schema and rethinking our resource search strategy.

If you’ve ever wondered what it takes as a platform engineer to support access control on a multi-tenant system with millions of resources, this is your opportunity to learn how to orchestrate a migration from your current access control system and hear about the peculiar challenges of developing security critical systems.