How JPMorgan Chase uses Grafana and AI to monitor SLOs, SLIs, and more
For the team at JPMorgan Chase, the daily stakes of having a stable system are high.
“We are in the business of making sure that trades are executed, and systems are stable and up and running for a positive client experience,” said Askari Imam, VP, Asset Wealth Management (Product and Integration Delivery). “But when things go wrong, things really went wrong.”
In their recent ObservabilityCon 2022 session titled “Monitoring on steroids: How JPMorgan Chase uses Grafana for their trading platform to spot issues quickly and proactively,” Imam, Product Owner Crystal Sorensen, and Rich Mirsberger, VP, Global Technology, traced their path to building a new tool for proactive monitoring.
Walking through examples using Tesla and Meme stocks, Sorensen demonstrated the failures (and the financial impact due to failed trades) that can occur across a complex order execution system lacking centralized monitoring.
“We couldn’t see how things correlated together and how one thing affected another. We needed something measurable and reliable,” Sorensen said. They wanted something that not only allowed them to find and fix the problem quickly, but also anticipate issues.
Having Grafana as part of their new observability solution was a no-brainer. “We liked the open telemetry we get from the Grafana tool set. Because who wants to start over, right?” said Sorensen. “You have things in place. You want to bring those things together and you want them to work and you want them to work well.”
Leveling up observability: “We are so far ahead now”
In March 2020, the JPMC team was at a crossroads. There was volatility in the market, the team went remote due to the pandemic, and the company’s online presence was growing. Plus, they had institutional clients with complicated algorithms and specific needs.
“In our previous life, we were reactive. We were dealing with alerts. We needed to scale,” says Imam. The team looked for ways to be more proactive when monitoring their technical landscape.
Their solution: marrying their existing, robust order management system with modern tool sets. Using Prometheus for data scraping and Grafana as a visualization tool, “you get a beautiful bundle of joy that we call TradeMon.”
The new solution has unlocked real-time data, such as CPU memory and volume, that gives the team better insight into capacity and potential volatility. Imam also noted that Grafana’s visualization allows them to build a more transparent community thanks to easy-to-read and shareable reports anyone from the C-suite to the trading desk can access and decipher.
“We are so far ahead now,” says Imam.
Next up: leveraging AI
Walking through their good and bad event modeling, Rich Mirsberger, VP of Global Technology, shared their initial partnership with the AIOps team. “When we don’t meet our SLO [service level objective], there is accountability,” says Mirsrberger, adding that bad events include unexpected behavior like a system outage. Together, the two teams took another look at the error budget and threshold graphs to ensure they had the most accurate dynamic model possible. The result was a success: Both the AI-provided line and Mirsberger’s team’s manual model fell within 1 percentage point of one another.
Now, they are expanding that effort elsewhere. They plan on instituting more AIOps and machine learning to automatically identify new SLOs and market volatility anomalies. They also want coverage during non-market hours and to add report creation tools. Said Sorensen: “Everyone wants bigger, better, faster, more.”
To get an inside look at how JPMorgan Chase’s custom observability tool works, watch the JPMC team’s full session. All our sessions from ObservabilityCON 2022 are now available on demand.
