AI/MLCXOsNewsOpen Source

Impact Of Visual AI On Test Automation Shows 5.9x Better Test Code Efficiency


Visual AI powered test automation company Applitools has announced an industry report that reveals the impact of Visual AI on the performance of the most popular open source test frameworks. These include Cypress, Selenium, and WebdriverIO.

According to the company, data was sourced from 288 quality engineers across 101 countries thus spending an average of 11 hours during November 2019. This amounts to a combined 80 work weeks, or well over a year and a half of quality engineering hours.

  • 5.8x Faster Test Creation: When using existing open source code-based frameworks in isolation, it took testers 7 hours to create test code in contrast to just 1.2 hours using Visual AI.
  • 3.8x Increase in Test Code Stability: Participants used 32 locators and labels on average to provide test coverage using code based frameworks alone. This number dropped to just 7 with Visual AI. This increased stability greatly reduces test flakiness, broken CI/CD builds, and time spent on maintenance.
  • 45% More Efficient at Catching Bugs: Participants caught 19 out of 20 bugs (95%) using Visual AI compared to just 13 (65%) when using code based frameworks in isolation.
  • All 288 testers that participated in the study achieved success rates of 79% when using their preferred code-based framework vs. 88% with Visual AI, the report added.
  • This is a nine point jump in their performance after just a single hour of Visual AI training, according to the company.

“Visual AI is a tremendous efficiency multiplier for automated testing,” said Gil Sever, CEO of Applitools. “Testers need to look at Visual AI as a superpower. A virtual assistant that can access every screen and page in minutes, inspect it for changes and interpret every one of them the same way the human eye and brain would, but do so without ever getting tired or making any mistakes. We believe this capability is a major part of the future of test automation.”