In this video, Xiaofeng and Jialin Jiao from Roblox introduce their journey in building a robust runtime diagnostic system using Pyroscope. With over 70 million daily active users and 4.4 million creators contributing to the platform, ensuring reliability and efficiency is paramount. They discuss the challenges faced in debugging production issues and the manual, inefficient methods previously used. Through thorough investigation and collaboration with Grafana Labs, they developed an on-demand profiling workflow, enabling engineers to identify and address performance bottlenecks effectively. They share success stories where this system led to significant CPU reductions and throughput improvements. The ultimate goal is to integrate this system seamlessly into their toolchain, ensuring continuous improvement and uninterrupted experiences for Roblox users.

Xiaofeng Han

Head of Observability, Roblox

Xiaofeng Han is the head of observability at Roblox. His team works on a wide range of monitoring, alerting, debugging and testing infrastructures. He is very passionate about bringing intelligence into the observability domain to deliver the full potential and insights of the observability data. Before Roblox, Xiaofeng had been working with Google for 10 years where he led the tools and infrastructure teams to support Google SearchAds org. Xiaofeng obtained his Ph.D. degree from the University of Delaware in Computer Science. He authored and co-authored more than 10 highly referenced research papers on engineer efficiency and optimization in wireless networks. Xiaofeng is currently living in San Jose with his wife and 15-year old son Grant. He enjoyed various outdoor events and building robots with his son.

Jialin Jiao

Principal Engineer and Engineering Manager, Roblox

Jialin is currently a Principal Engineer and Engineering Manager within Roblox’s Observability team. His scope includes using tools like distributed tracing, profiling and memory dump to improve the efficiency and experience of debugging and optimizing production services in Roblox. He is an ex-Microsoft, ex-Uber and ex-Meta engineer with more than 10 years of experience in platform, ML infrastructure and product development.