Carvana Speeds Up Test Creation by 100% with Grafana k6

Key takeaways

With k6 Cloud Enterprise, load and performance testing has become more approachable and reliable for Carvana, and trust in testing has significantly increased.
k6 is based on JavaScript, so Carvana’s developers can quickly write complex load tests.
k6 Cloud Enterprise reporting and graphical features instill confidence in the test results and capabilities of Carvana’s applications.

-–

Carvana is a leading e-commerce platform for buying and selling used vehicles, offering as-soon-as-next-day delivery in 300+ markets across the U.S. and pick-up at 28 patented signature car vending machines.

Consumers can browse through more than 45,000 vehicles on carvana.com, then purchase and finance a quality used vehicle in as little as five minutes- all from the comfort and safety of home or on the go via a mobile device. Carvana vehicles come with a 7-day return policy, have undergone a rigorous, 150-point inspection, and have never been in a reported accident.

The challenge

For years, the platform development and engineering teams at Carvana used Microsoft Visual Studio for load and performance testing. But when Microsoft announced that the company was sunsetting Visual Studio’s load testing features, the teams realized they needed to immediately start searching for an alternative vendor.

The quality engineering (QE) team needed a load and performance testing tool that would allow them to develop tests that accurately reflect spikes and increased user load during Carvana’s busiest times, especially the holiday season. The team also wanted a tool that Carvana’s developers would feel comfortable using and would measure their progress before each release.

The QE team spent months evaluating GUI-based and code-based tools. They created a proof of concept for each tool and found that each one came up short in one or more areas. For example, the QE team wanted to run complicated test cases, but the GUI-based tools they evaluated lacked in test complexity and customization. And the code-based tools they reviewed were too complex, so developers couldn’t onboard quickly.

The solution

The QE team built a framework that allows Carvana’s engineers and developers to quickly scaffold tests for the various applications they are charged with testing. The framework allows them to reuse part of the code for functional, load, and performance testing, making it easy to work across teams.

The team built the framework on top of the k6 JavaScript API and also incorporated k6 Cloud Enterprise. k6 is based on JavaScript, so Carvana’s developers feel comfortable using the tool and can quickly write complex load tests. The engineering and development teams use k6 Cloud Enterprise scheduled tests to automate some of the performance testing process and scenarios to load test services more accurately.

The QE team uses the reporting and graphical features to see load tests execute in real time and drill into test results, instilling confidence in the test results and capabilities of Carvana’s applications. Non-coders use k6 to review the performance of applications undergoing testing and make informed decisions.

Having a JavaScript-based tool allows the authoring of complex tests to happen quickly, and when combined with the rich analytics of k6 Cloud Enterprise, our developers and QA team get a powerful toolset to test applications and make informed decisions.
Tyler Monteith, Senior Quality Automation Engineer, Carvana

The results

Thanks to the new framework, the engineering and development teams follow a continuous performance testing philosophy that allows them to build better applications and deliver a first-class user experience. And reusing custom libraries has significantly improved team efficiency as the number of tests grows and applications change. The teams are now more dynamic and agile when it comes to Carvana’s needs during the e-commerce holiday season and overall business goals. The QE team has seen overwhelmingly positive results in developer participation, leadership confidence, and testing turnaround.

k6 Cloud Enterprise has enabled our quality engineering team to build more confident testing and streamline the process for deployment of new features and products, creating an all-around first-class experience for our customers.
Eric Stone, Quality Engineer II, Carvana